[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:01:46 dist-test-slave-6c7s krb5kdc[17154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:01:46 dist-test-slave-6c7s krb5kdc[17154](info): set up 2 sockets
Sep 27 21:01:46 dist-test-slave-6c7s krb5kdc[17154](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 21:01:47.917606 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:01:48.052414 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:41123
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:41123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:48.135530 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:48.135749 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:48.135773 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:48.136919 17167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:01:48.136957 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:48.136970 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:48.136983 17167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:01:48.136996 17167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:01:48.138438 17167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:41123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:41123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:48.138625 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:48.138826 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:48.144661 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.146274 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.146332 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:48.146590 17167 server_base.cc:1034] running on GCE node
I20260927 21:01:48.146888 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:48.147219 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:48.148389 17167 hybrid_clock.cc:648] HybridClock initialized: now 1790542908148376 us; error 32 us; skew 500 ppm
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:48.151017 17167 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:01:48.151513 17167 webserver.cc:460] Webserver started at http://127.16.190.254:36183/ using document root <none> and password file <none>
I20260927 21:01:48.151770 17167 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:48.151839 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:48.151988 17167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:48.153247 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "ee0d9fc76ac7493d85d77d2e4811e125"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "675acd7a83ead1c9a521c2d3d4745fbf"
server_key_iv: "6808cb3411c04199eecc6d945543fdef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.153731 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "ee0d9fc76ac7493d85d77d2e4811e125"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "675acd7a83ead1c9a521c2d3d4745fbf"
server_key_iv: "6808cb3411c04199eecc6d945543fdef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.155185 17167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:01:48.155982 17182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.156186 17167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:48.156301 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "ee0d9fc76ac7493d85d77d2e4811e125"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "675acd7a83ead1c9a521c2d3d4745fbf"
server_key_iv: "6808cb3411c04199eecc6d945543fdef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.156380 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:48.175048 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:48.177567 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:48.177759 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:48.181792 17167 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:41123
I20260927 21:01:48.181843 17234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:41123 every 8 connection(s)
I20260927 21:01:48.182236 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:01:48.183086 17235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:48.185421 17235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125: Bootstrap starting.
I20260927 21:01:48.186030 17235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:48.186326 17235 log.cc:824] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:48.187072 17235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125: No bootstrap required, opened a new log
I20260927 21:01:48.187929 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17167
I20260927 21:01:48.188011 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:01:48.188656 17147 external_mini_cluster.cc:1428] Setting key 4d70e750a9c0fbe38f0be8f9fe5e7595
I20260927 21:01:48.188863 17235 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } }
I20260927 21:01:48.189026 17235 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:48.189072 17235 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d9fc76ac7493d85d77d2e4811e125, State: Initialized, Role: FOLLOWER
I20260927 21:01:48.189246 17235 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } }
I20260927 21:01:48.189348 17235 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:01:48.189394 17235 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:01:48.189450 17235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:48.190433 17235 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } }
I20260927 21:01:48.190554 17235 leader_election.cc:302] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0d9fc76ac7493d85d77d2e4811e125; no voters:
I20260927 21:01:48.190706 17235 leader_election.cc:288] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:01:48.190737 17240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:48.190843 17240 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [term 1 LEADER]: Becoming Leader. State: Replica: ee0d9fc76ac7493d85d77d2e4811e125, State: Running, Role: LEADER
I20260927 21:01:48.190922 17240 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } }
I20260927 21:01:48.190958 17235 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:01:48.191219 17240 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0d9fc76ac7493d85d77d2e4811e125. Latest consensus state: current_term: 1 leader_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } } }
I20260927 21:01:48.191298 17240 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:48.191229 17242 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0d9fc76ac7493d85d77d2e4811e125" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41123 } } }
I20260927 21:01:48.191674 17242 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125 [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:48.191952 17247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:01:48.192602 17247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:01:48.194491 17247 catalog_manager.cc:1347] Generated new cluster ID: d270a1668e714c3f9b4e0e052df49bd4
I20260927 21:01:48.194543 17247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:48.203912 17247 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:01:48.204663 17247 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:01:48.208012 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.189577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.16.190.254:41123)
0927 21:01:48.189803 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.189813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.190077 (+ 264us) server_negotiation.cc:407] Connection header received
0927 21:01:48.192363 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.192372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.192448 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.192555 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.195134 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.196876 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.198383 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.199133 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.199261 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.203826 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.203831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.203832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.203845 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.205982 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.206483 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.206486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.206487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.206605 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.206915 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.206915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.206916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.207153 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.207318 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.207597 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.207757 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260927 21:01:48.216734 17247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee0d9fc76ac7493d85d77d2e4811e125: Generated new TSK 0
I20260927 21:01:48.217006 17247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:01:48.248973 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:48.331264 17261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:48.331476 17261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:48.331501 17261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:48.332662 17261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:48.332707 17261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:48.332736 17261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:01:48.334312 17261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:48.334584 17261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:48.334844 17261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:48.339958 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.339958 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.340018 17266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:48.340200 17261 server_base.cc:1034] running on GCE node
I20260927 21:01:48.340616 17261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:48.340889 17261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:48.342077 17261 hybrid_clock.cc:648] HybridClock initialized: now 1790542908342056 us; error 36 us; skew 500 ppm
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:48.344872 17261 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:01:48.345323 17261 webserver.cc:460] Webserver started at http://127.16.190.193:35757/ using document root <none> and password file <none>
I20260927 21:01:48.345543 17261 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:48.345594 17261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:48.345752 17261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:48.346853 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "61355aae91114f7eaf16844ae5719e29"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "fc348c755290050a5e4acf8bcb7fff1a"
server_key_iv: "e2199301c3f7a1a4dcc3f0ce31f21ab6"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.347223 17261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "61355aae91114f7eaf16844ae5719e29"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "fc348c755290050a5e4acf8bcb7fff1a"
server_key_iv: "e2199301c3f7a1a4dcc3f0ce31f21ab6"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.348621 17261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:01:48.349478 17276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.349709 17261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:01:48.349804 17261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "61355aae91114f7eaf16844ae5719e29"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "fc348c755290050a5e4acf8bcb7fff1a"
server_key_iv: "e2199301c3f7a1a4dcc3f0ce31f21ab6"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.349884 17261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:48.364511 17261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:48.367071 17261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:48.367221 17261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:48.367494 17261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:48.367822 17261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:48.367856 17261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.367878 17261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:48.367923 17261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.374018 17261 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:45641
I20260927 21:01:48.374204 17389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:45641 every 8 connection(s)
I20260927 21:01:48.374475 17261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:48.384086 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17261
I20260927 21:01:48.384169 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:01:48.384337 17147 external_mini_cluster.cc:1428] Setting key d61ea65f78ba2f207460e5a1e155d530
I20260927 21:01:48.385711 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.375298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41123 (local address 127.16.190.193:48087)
0927 21:01:48.375488 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:48.375560 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:48.376385 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:48.376683 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:48.376686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:48.377474 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:48.377704 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.377710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.378478 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.378479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.379319 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.379321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.379902 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.379905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.380042 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.380575 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:48.380585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:48.382211 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:48.383955 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.383962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.383964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.384317 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.384608 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.384609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.384609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.384789 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.385109 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:48.385117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:48.385292 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 21:01:48.385381 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:01:48.385523 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260927 21:01:48.385705 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.375335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:48087 (local address 127.16.190.254:41123)
0927 21:01:48.375502 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.375505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.375685 (+ 180us) server_negotiation.cc:407] Connection header received
0927 21:01:48.376534 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.376536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.376548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.376614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.377838 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.378376 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.379427 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.379689 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.379759 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.382327 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.382329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.382329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.382337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.383856 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.384399 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.384400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.384401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.384474 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.384894 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.384895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.384896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.385036 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.385104 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.385456 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.385561 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260927 21:01:48.386524 17390 heartbeater.cc:347] Connected to a master server at 127.16.190.254:41123
I20260927 21:01:48.386651 17390 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:48.386880 17390 heartbeater.cc:511] Master 127.16.190.254:41123 requested a full tablet report, sending...
I20260927 21:01:48.387429 17199 ts_manager.cc:194] Registered new tserver with Master: 61355aae91114f7eaf16844ae5719e29 (127.16.190.193:45641)
I20260927 21:01:48.388051 17199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:48087
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:01:48.412708 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:48.495797 17395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:48.496021 17395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:48.496047 17395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:48.497305 17395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:48.497354 17395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:48.497382 17395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:01:48.498894 17395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:48.499171 17395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:48.499428 17395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:48.504801 17401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.504851 17403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.504801 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:48.505256 17395 server_base.cc:1034] running on GCE node
I20260927 21:01:48.505501 17395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:48.505846 17395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:48.507063 17395 hybrid_clock.cc:648] HybridClock initialized: now 1790542908507038 us; error 33 us; skew 500 ppm
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:48.509881 17395 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:01:48.510365 17395 webserver.cc:460] Webserver started at http://127.16.190.194:43009/ using document root <none> and password file <none>
I20260927 21:01:48.510608 17395 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:48.510697 17395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:48.510847 17395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:48.511986 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "b5d175be19ce4af1af9a14fb377979b8"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "12949a1a36209f4c111ae0821469639e"
server_key_iv: "6dacc8edadec18142b0ad469df4137ef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.512367 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "b5d175be19ce4af1af9a14fb377979b8"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "12949a1a36209f4c111ae0821469639e"
server_key_iv: "6dacc8edadec18142b0ad469df4137ef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.513892 17395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:01:48.514743 17410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.514950 17395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:01:48.515066 17395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "b5d175be19ce4af1af9a14fb377979b8"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "12949a1a36209f4c111ae0821469639e"
server_key_iv: "6dacc8edadec18142b0ad469df4137ef"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.515151 17395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:48.525502 17395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:48.528187 17395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:48.528364 17395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:48.528620 17395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:48.529011 17395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:48.529067 17395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.529104 17395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:48.529131 17395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.534865 17395 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:42207
I20260927 21:01:48.534945 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:42207 every 8 connection(s)
I20260927 21:01:48.535283 17395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:01:48.537604 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17395
I20260927 21:01:48.537679 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:01:48.537854 17147 external_mini_cluster.cc:1428] Setting key 38beb0301c0ab5663b30caa83e4349b4
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:48.546552 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.536034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:42803 (local address 127.16.190.254:41123)
0927 21:01:48.536122 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.536125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.536363 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:01:48.536979 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.536981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.536992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.537056 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.538279 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.538995 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.540298 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.540542 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.540602 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.543336 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.543337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.543338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.543348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.545067 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.545495 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.545497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.545497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.545562 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.545815 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.545816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.545817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.545979 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.546057 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.546324 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.546419 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:01:48.546584 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.535993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41123 (local address 127.16.190.194:42803)
0927 21:01:48.536204 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:48.536257 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:48.536787 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:48.537116 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:48.537119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:48.537835 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:48.538170 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.538177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.539120 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.539121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.540161 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.540164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.540634 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.540635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.540791 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.541443 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:48.541452 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:48.543229 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:48.545154 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.545160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.545163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.545413 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.545632 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.545633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.545633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.545745 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.546050 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:48.546054 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:48.546179 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 21:01:48.546273 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:01:48.546407 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
I20260927 21:01:48.547118 17524 heartbeater.cc:347] Connected to a master server at 127.16.190.254:41123
I20260927 21:01:48.547235 17524 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:48.547447 17524 heartbeater.cc:511] Master 127.16.190.254:41123 requested a full tablet report, sending...
I20260927 21:01:48.547906 17199 ts_manager.cc:194] Registered new tserver with Master: b5d175be19ce4af1af9a14fb377979b8 (127.16.190.194:42207)
I20260927 21:01:48.548400 17199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:42803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:01:48.568391 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:48.651046 17529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:48.651259 17529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:48.651283 17529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:48.652586 17529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:48.652634 17529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:48.652663 17529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:01:48.654452 17529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:41123
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:48.654727 17529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:48.655010 17529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:48.660414 17534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.660396 17537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:48.660396 17535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:48.660703 17529 server_base.cc:1034] running on GCE node
I20260927 21:01:48.661065 17529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:48.661383 17529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:48.662552 17529 hybrid_clock.cc:648] HybridClock initialized: now 1790542908662528 us; error 38 us; skew 500 ppm
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:48.665323 17529 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:01:48.665802 17529 webserver.cc:460] Webserver started at http://127.16.190.195:42045/ using document root <none> and password file <none>
I20260927 21:01:48.666081 17529 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:48.666157 17529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:48.666318 17529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:48.667430 17529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "3cdd7834a9f9498493a4a1425b473c3f"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "757121419b61620ad64d1460100f1447"
server_key_iv: "c8189d22113019df37a60a4d6d14f0df"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.667815 17529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "3cdd7834a9f9498493a4a1425b473c3f"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "757121419b61620ad64d1460100f1447"
server_key_iv: "c8189d22113019df37a60a4d6d14f0df"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.669330 17529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:01:48.670159 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.670372 17529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:48.670472 17529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3cdd7834a9f9498493a4a1425b473c3f"
format_stamp: "Formatted at 2026-09-27 21:01:48 on dist-test-slave-6c7s"
server_key: "757121419b61620ad64d1460100f1447"
server_key_iv: "c8189d22113019df37a60a4d6d14f0df"
server_key_version: "encryptionkey@0"
I20260927 21:01:48.670560 17529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:48.684691 17529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:48.687152 17529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:48.687292 17529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:48.687474 17529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:48.687803 17529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:48.687852 17529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.687875 17529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:48.687888 17529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:48.694216 17529 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:45125
I20260927 21:01:48.694331 17657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:45125 every 8 connection(s)
I20260927 21:01:48.694648 17529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:48.703923 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17529
I20260927 21:01:48.704034 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:01:48.704190 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.695397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41123 (local address 127.16.190.195:57895)
0927 21:01:48.695539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:48.695577 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:48.696084 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:48.696295 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:48.696298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:48.696967 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:48.697170 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.697175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.697855 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.697856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.698651 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:48.698652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.699002 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.699003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:48.699115 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.699683 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:48.699691 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:48.701115 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:48.702680 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.702688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.702690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.702931 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.703131 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:48.703132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:48.703132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:48.703264 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:48.703541 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:48.703547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:48.703683 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:01:48.703787 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:01:48.703922 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260927 21:01:48.704275 17147 external_mini_cluster.cc:1428] Setting key 5f5b0b6bb14b4820fc673e4a3a253e6d
I20260927 21:01:48.704350 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.695407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:57895 (local address 127.16.190.254:41123)
0927 21:01:48.695470 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.695472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.695665 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:01:48.696170 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.696171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.696181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.696231 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.697256 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.697774 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.698746 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.698930 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.698982 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.701205 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.701206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.701207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.701214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.702582 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.703006 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.703007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.703007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.703062 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.703332 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.703333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.703333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.703451 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.703506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.704164 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.704263 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:01:48.704851 17658 heartbeater.cc:347] Connected to a master server at 127.16.190.254:41123
I20260927 21:01:48.704975 17658 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:48.705188 17658 heartbeater.cc:511] Master 127.16.190.254:41123 requested a full tablet report, sending...
I20260927 21:01:48.705576 17199 ts_manager.cc:194] Registered new tserver with Master: 3cdd7834a9f9498493a4a1425b473c3f (127.16.190.195:45125)
I20260927 21:01:48.705928 17199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:57895
I20260927 21:01:48.716435 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:48.739863 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.731935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.16.190.193:45641)
0927 21:01:48.732100 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.732110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.732121 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:01:48.732163 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.732165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.732179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.732253 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.732790 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.733560 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.734255 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.734571 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.734626 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.737086 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.737099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.737103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.737111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.738585 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.738970 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.738974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.738976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.739038 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.739256 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.739257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.739257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.739409 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.739599 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.739646 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.739732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260927 21:01:48.740285 17304 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44314
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:48.762894 17665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:48.755772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.16.190.193:45641)
0927 21:01:48.755851 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:48.755853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:48.755865 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:01:48.755921 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:48.755922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:48.755934 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:48.755992 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:48.756567 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.757159 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.757907 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:48.758114 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:48.758172 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:48.760578 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:48.760580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:48.760580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:48.760586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:48.761925 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.762240 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.762241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.762241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.762287 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:48.762483 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:48.762483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:48.762483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:48.762569 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:48.762691 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:48.762768 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:01:48.762828 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:01:48.763468 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17261
I20260927 21:01:48.768201 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17395
I20260927 21:01:48.773526 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17529
I20260927 21:01:48.777990 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17167
2026-09-27T21:01:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1942 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17672](info): set up 2 sockets
Sep 27 21:01:48 dist-test-slave-6c7s krb5kdc[17672](info): commencing operation
krb5kdc: starting...
W20260927 21:01:49.817108 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:01:49 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:01:49.939800 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38041
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:38041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:50.023241 17685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:50.023495 17685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:50.023537 17685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:50.025650 17685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:01:50.025708 17685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:50.025728 17685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:50.025746 17685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:01:50.025763 17685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:01:50.028020 17685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:38041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:50.028311 17685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:50.028604 17685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:50.034190 17691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.034199 17690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.034418 17685 server_base.cc:1034] running on GCE node
W20260927 21:01:50.034204 17693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.034855 17685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:50.035151 17685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:50.036296 17685 hybrid_clock.cc:648] HybridClock initialized: now 1790542910036278 us; error 37 us; skew 500 ppm
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:50.038838 17685 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:01:50.039264 17685 webserver.cc:460] Webserver started at http://127.16.190.254:45215/ using document root <none> and password file <none>
I20260927 21:01:50.039486 17685 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:50.039556 17685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:50.039707 17685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:50.040802 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "1dcd5a5ba87245c983fb19979b5e341a"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "3478b162dd0658815929a76785f70035"
server_key_iv: "c71922f612e5fe24140452d69fffb963"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.041239 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "1dcd5a5ba87245c983fb19979b5e341a"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "3478b162dd0658815929a76785f70035"
server_key_iv: "c71922f612e5fe24140452d69fffb963"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.042573 17685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:50.043391 17700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.043565 17685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:50.043661 17685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "1dcd5a5ba87245c983fb19979b5e341a"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "3478b162dd0658815929a76785f70035"
server_key_iv: "c71922f612e5fe24140452d69fffb963"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.043751 17685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:50.059288 17685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:50.061681 17685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:50.061882 17685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:50.066111 17685 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:38041
I20260927 21:01:50.066154 17752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:38041 every 8 connection(s)
I20260927 21:01:50.066512 17685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:01:50.067415 17753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:50.069766 17753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a: Bootstrap starting.
I20260927 21:01:50.070312 17753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:50.070540 17753 log.cc:824] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:50.071244 17753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a: No bootstrap required, opened a new log
I20260927 21:01:50.073062 17753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } }
I20260927 21:01:50.073210 17753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:50.073233 17753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcd5a5ba87245c983fb19979b5e341a, State: Initialized, Role: FOLLOWER
I20260927 21:01:50.073374 17753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } }
I20260927 21:01:50.073451 17753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:01:50.073474 17753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:01:50.073503 17753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:50.074013 17753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } }
I20260927 21:01:50.074096 17753 leader_election.cc:302] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dcd5a5ba87245c983fb19979b5e341a; no voters:
I20260927 21:01:50.074225 17753 leader_election.cc:288] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:01:50.074296 17756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:50.074465 17756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [term 1 LEADER]: Becoming Leader. State: Replica: 1dcd5a5ba87245c983fb19979b5e341a, State: Running, Role: LEADER
I20260927 21:01:50.074502 17753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:01:50.074565 17756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } }
I20260927 21:01:50.075112 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17685
I20260927 21:01:50.075207 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:01:50.075299 17758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dcd5a5ba87245c983fb19979b5e341a. Latest consensus state: current_term: 1 leader_uuid: "1dcd5a5ba87245c983fb19979b5e341a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } } }
I20260927 21:01:50.075392 17758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:50.075383 17147 external_mini_cluster.cc:1428] Setting key 1e529b48f72c72ab73038d4dafdd2a1f
I20260927 21:01:50.075688 17757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dcd5a5ba87245c983fb19979b5e341a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dcd5a5ba87245c983fb19979b5e341a" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38041 } } }
I20260927 21:01:50.075784 17757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:50.075877 17765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:01:50.076440 17765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:01:50.079075 17765 catalog_manager.cc:1347] Generated new cluster ID: bd6661cef98d4b76add390ac42e1b46a
I20260927 21:01:50.079160 17765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:50.086113 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.076078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.16.190.254:38041)
0927 21:01:50.076258 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.076380 (+ 122us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.076417 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:01:50.076574 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.076582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.076639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.076792 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.077540 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.078936 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.079785 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.080217 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.080333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.082686 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.082690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.082690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.082702 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.084411 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.084783 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.084786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.084786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.084952 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.085162 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.085163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.085163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.085431 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.085603 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.085687 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.085937 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260927 21:01:50.088423 17765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:01:50.088999 17765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:01:50.093544 17765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dcd5a5ba87245c983fb19979b5e341a: Generated new TSK 0
I20260927 21:01:50.093734 17765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:01:50.124291 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:50.207774 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:50.208048 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:50.208093 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:50.209410 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:50.209478 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:50.209528 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:01:50.211184 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:50.211483 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:50.211733 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:50.217031 17787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.217031 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.217056 17784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.217247 17779 server_base.cc:1034] running on GCE node
I20260927 21:01:50.217525 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:50.217739 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:50.218923 17779 hybrid_clock.cc:648] HybridClock initialized: now 1790542910218896 us; error 38 us; skew 500 ppm
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:50.221951 17779 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:01:50.222446 17779 webserver.cc:460] Webserver started at http://127.16.190.193:39245/ using document root <none> and password file <none>
I20260927 21:01:50.222695 17779 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:50.222764 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:50.222915 17779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:50.223991 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "e10a3f185aa8471fa1fc050d202f55a6"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "a417275fa6265160885e2a6eb5471526"
server_key_iv: "a7dc585a9051bc623c58d0bb2f82572f"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.224371 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "e10a3f185aa8471fa1fc050d202f55a6"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "a417275fa6265160885e2a6eb5471526"
server_key_iv: "a7dc585a9051bc623c58d0bb2f82572f"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.225826 17779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:01:50.226725 17794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.227042 17779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:01:50.227175 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "e10a3f185aa8471fa1fc050d202f55a6"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "a417275fa6265160885e2a6eb5471526"
server_key_iv: "a7dc585a9051bc623c58d0bb2f82572f"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.227267 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:50.238883 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:50.241197 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:50.241392 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:50.241627 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:50.241935 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:50.241986 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.242027 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:50.242054 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.248018 17779 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42169
I20260927 21:01:50.248055 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42169 every 8 connection(s)
I20260927 21:01:50.248390 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:01:50.249377 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17779
I20260927 21:01:50.249475 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:01:50.249645 17147 external_mini_cluster.cc:1428] Setting key 8e3d0d758c0c7b4aa27400449f6d3f0c
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:01:50.260546 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.249166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:56941 (local address 127.16.190.254:38041)
0927 21:01:50.249243 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.249246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.249457 (+ 211us) server_negotiation.cc:407] Connection header received
0927 21:01:50.250837 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.250839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.250855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.250923 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.251743 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.252311 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.253521 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.253894 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.253969 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.256966 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.256968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.256968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.256979 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.258968 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.259410 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.259412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.259413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.259479 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.259764 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.259765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.259765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.259925 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.260004 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.260314 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.260419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:01:50.260732 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.249123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38041 (local address 127.16.190.193:56941)
0927 21:01:50.249302 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.249350 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.250722 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.251004 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.251007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.251372 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:50.251629 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.251634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.252417 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.252418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.253415 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.253418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.254008 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.254009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.254155 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.254781 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:50.254790 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:50.256775 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:50.259067 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.259073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.259075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.259324 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.259575 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.259576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.259576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.259689 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.259998 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:50.260003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:50.260133 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.260273 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.260412 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260927 21:01:50.261391 17908 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38041
I20260927 21:01:50.261509 17908 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:50.261677 17908 heartbeater.cc:511] Master 127.16.190.254:38041 requested a full tablet report, sending...
I20260927 21:01:50.262247 17717 ts_manager.cc:194] Registered new tserver with Master: e10a3f185aa8471fa1fc050d202f55a6 (127.16.190.193:42169)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:01:50.263149 17717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:56941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:01:50.280323 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:50.361181 17913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:50.361392 17913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:50.361415 17913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:50.362510 17913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:50.362545 17913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:50.362571 17913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:01:50.363969 17913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:50.364161 17913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:50.364332 17913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:50.369510 17921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.369537 17919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.369623 17913 server_base.cc:1034] running on GCE node
W20260927 21:01:50.369521 17918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.369997 17913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:50.370250 17913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:50.371416 17913 hybrid_clock.cc:648] HybridClock initialized: now 1790542910371410 us; error 37 us; skew 500 ppm
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:50.374025 17913 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:01:50.374497 17913 webserver.cc:460] Webserver started at http://127.16.190.194:43901/ using document root <none> and password file <none>
I20260927 21:01:50.374737 17913 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:50.374805 17913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:50.374939 17913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:50.375994 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "53dd194cc1634e219d0910642ab628da"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "21db419eea15d74eea9d007a8036a5ba"
server_key_iv: "f76f16d986a3c744e85af7f4d0688a73"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.376415 17913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "53dd194cc1634e219d0910642ab628da"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "21db419eea15d74eea9d007a8036a5ba"
server_key_iv: "f76f16d986a3c744e85af7f4d0688a73"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.377848 17913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:01:50.378582 17928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.378750 17913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:50.378850 17913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "53dd194cc1634e219d0910642ab628da"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "21db419eea15d74eea9d007a8036a5ba"
server_key_iv: "f76f16d986a3c744e85af7f4d0688a73"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.378932 17913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:50.397651 17913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:50.399802 17913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:50.399974 17913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:50.400228 17913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:50.400548 17913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:50.400604 17913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.400645 17913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:50.400674 17913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.406442 17913 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:38265
I20260927 21:01:50.406486 18041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:38265 every 8 connection(s)
I20260927 21:01:50.406821 17913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:50.415438 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 17913
I20260927 21:01:50.415539 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:01:50.415701 17147 external_mini_cluster.cc:1428] Setting key 0bf16bb4c03ffd64c0b72a50aa1c8f90
I20260927 21:01:50.416225 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.407611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43037 (local address 127.16.190.254:38041)
0927 21:01:50.407705 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.407708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.407900 (+ 192us) server_negotiation.cc:407] Connection header received
0927 21:01:50.408936 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.408937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.408952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.408999 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.409677 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.410185 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.411230 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.411430 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.411522 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.413644 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.413646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.413646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.413653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.414858 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.415211 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.415211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.415212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.415266 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.415558 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.415559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.415560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.415691 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.415769 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.416088 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.416162 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:01:50.416493 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.407586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38041 (local address 127.16.190.194:43037)
0927 21:01:50.407763 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.407811 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.408766 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.409086 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.409090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.409377 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:50.409570 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.409575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.410287 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.410289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.411131 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.411134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.411542 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.411543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.411669 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.412153 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:50.412161 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:50.413558 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:50.414929 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.414935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.414936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.415148 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.415329 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.415330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.415331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.415473 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.415764 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:50.415771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:50.415936 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.416061 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.416231 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260927 21:01:50.416989 18042 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38041
I20260927 21:01:50.417100 18042 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:50.417279 18042 heartbeater.cc:511] Master 127.16.190.254:38041 requested a full tablet report, sending...
I20260927 21:01:50.417713 17717 ts_manager.cc:194] Registered new tserver with Master: 53dd194cc1634e219d0910642ab628da (127.16.190.194:38265)
I20260927 21:01:50.418099 17717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:43037
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:01:50.442322 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:50.522774 18047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:50.522990 18047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:50.523015 18047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:50.524094 18047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:50.524129 18047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:50.524155 18047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:01:50.525579 18047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:50.525760 18047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:50.525930 18047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:50.530901 18053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.530906 18052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:50.530911 18055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:50.531211 18047 server_base.cc:1034] running on GCE node
I20260927 21:01:50.531440 18047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:50.531685 18047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:50.532896 18047 hybrid_clock.cc:648] HybridClock initialized: now 1790542910532878 us; error 29 us; skew 500 ppm
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:50.535408 18047 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:01:50.535871 18047 webserver.cc:460] Webserver started at http://127.16.190.195:34563/ using document root <none> and password file <none>
I20260927 21:01:50.536079 18047 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:50.536141 18047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:50.536238 18047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:50.537371 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "38971215c9104990b2cda57f9fe56967"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "4e11c98e1863351f85b46eb6592f1b79"
server_key_iv: "28150d17fd1efeda85735e307aaf9b0d"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.537714 18047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "38971215c9104990b2cda57f9fe56967"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "4e11c98e1863351f85b46eb6592f1b79"
server_key_iv: "28150d17fd1efeda85735e307aaf9b0d"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.539098 18047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:01:50.539749 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.539901 18047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:50.539978 18047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "38971215c9104990b2cda57f9fe56967"
format_stamp: "Formatted at 2026-09-27 21:01:50 on dist-test-slave-6c7s"
server_key: "4e11c98e1863351f85b46eb6592f1b79"
server_key_iv: "28150d17fd1efeda85735e307aaf9b0d"
server_key_version: "encryptionkey@0"
I20260927 21:01:50.540031 18047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:50.556984 18047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:50.559278 18047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:50.559398 18047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:50.559572 18047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:01:50.559895 18047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:50.559928 18047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.559952 18047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:50.559999 18047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:50.565826 18047 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:44499
I20260927 21:01:50.565884 18175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:44499 every 8 connection(s)
I20260927 21:01:50.566217 18047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:01:50.567143 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18047
I20260927 21:01:50.567274 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:01:50.567473 17147 external_mini_cluster.cc:1428] Setting key 643be3a432491f35af9e449c73053153
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:50.576458 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.566992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:34751 (local address 127.16.190.254:38041)
0927 21:01:50.567060 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.567062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.567276 (+ 214us) server_negotiation.cc:407] Connection header received
0927 21:01:50.568512 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.568513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.568523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.568571 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.569288 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.569777 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.570761 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.570983 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.571095 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.573231 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.573233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.573233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.573240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.574890 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.575304 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.575306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.575306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.575364 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.575659 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.575660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.575660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.575792 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.575875 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.576267 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.576354 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:01:50.576682 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.566945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38041 (local address 127.16.190.195:34751)
0927 21:01:50.567135 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.567174 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.568437 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.568634 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.568637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.568999 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:50.569204 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.569209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.569857 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.569858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.570667 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.570669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.571102 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.571104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.571217 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.571751 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:50.571760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:50.573143 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:50.575005 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.575011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.575012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.575234 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.575440 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:50.575440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:50.575441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:50.575611 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:50.575879 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:50.575887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:50.576109 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.576229 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.576410 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260927 21:01:50.577190 18176 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38041
I20260927 21:01:50.577342 18176 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:50.577564 18176 heartbeater.cc:511] Master 127.16.190.254:38041 requested a full tablet report, sending...
I20260927 21:01:50.577914 17717 ts_manager.cc:194] Registered new tserver with Master: 38971215c9104990b2cda57f9fe56967 (127.16.190.195:44499)
I20260927 21:01:50.578332 17717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:34751
I20260927 21:01:50.579332 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:01:50.587148 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.581011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.16.190.254:38041)
0927 21:01:50.581090 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.581092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.581207 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:01:50.581274 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.581275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.581286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.581331 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.581972 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.582471 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.583265 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.583474 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.583557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.584729 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.584730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.584730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.584737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.585968 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.586324 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.586325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.586326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.586381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.586585 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.586586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.586587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.586749 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.586859 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.586961 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.587043 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:01:50.589347 17717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:50.590952 17717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:50.597605 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.16.190.195:44499)
0927 21:01:50.594030 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.594034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.594043 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:50.594081 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.594083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.594098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.594197 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:50.594433 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595460 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.596534 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597155 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597262 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597321 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.597468 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.597528 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":84,"threads_started":1}
I20260927 21:01:50.597596 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:44499 (local address 127.0.0.1:51858)
0927 21:01:50.593431 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.593471 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.593542 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.594316 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.594316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.594319 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:50.594362 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.594363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595538 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.595540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.596450 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.596453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597262 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.597340 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597373 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.597447 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.597502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260927 21:01:50.597827 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58302 (local address 127.16.190.194:38265)
0927 21:01:50.593714 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.593720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.593732 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:01:50.593774 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.593776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.593793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.593884 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:50.594284 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595031 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.596467 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597241 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597317 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597369 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.597678 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.597730 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":73,"threads_started":1}
I20260927 21:01:50.597851 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:38265 (local address 127.0.0.1:58302)
0927 21:01:50.593638 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.593646 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.593678 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.593897 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.593900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.593901 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:50.594200 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.594202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595145 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.595146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.596379 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.596382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597406 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.597473 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597477 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.597548 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.597589 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":65,"threads_started":1}
I20260927 21:01:50.598378 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.16.190.193:42169)
0927 21:01:50.593720 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.593724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.593733 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:50.593774 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.593776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.593793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.593884 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:50.594234 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595370 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.596678 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597386 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.598261 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.598305 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260927 21:01:50.598456 18110 tablet_service.cc:1467] Processing CreateTablet for tablet 3066982050774232b40c7abc6ff20e9b (DEFAULT_TABLE table=test-table [id=4805376939a943cbb5f579e2a09bd4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:50.598791 18110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066982050774232b40c7abc6ff20e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:50.598690 17976 tablet_service.cc:1467] Processing CreateTablet for tablet 3066982050774232b40c7abc6ff20e9b (DEFAULT_TABLE table=test-table [id=4805376939a943cbb5f579e2a09bd4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:50.599028 17976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066982050774232b40c7abc6ff20e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:50.599057 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.593258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42169 (local address 127.0.0.1:59192)
0927 21:01:50.593518 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.593528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.593568 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.594021 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.594022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.594024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:50.594077 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.594079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.595492 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.595493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.596600 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.596602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.597564 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.597565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.597617 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.597621 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.598763 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.598811 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":35,"thread_start_us":56,"threads_started":1}
I20260927 21:01:50.600090 17842 tablet_service.cc:1467] Processing CreateTablet for tablet 3066982050774232b40c7abc6ff20e9b (DEFAULT_TABLE table=test-table [id=4805376939a943cbb5f579e2a09bd4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:50.600415 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066982050774232b40c7abc6ff20e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:50.601943 18190 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Bootstrap starting.
I20260927 21:01:50.602299 18191 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Bootstrap starting.
I20260927 21:01:50.602612 18190 tablet_bootstrap.cc:654] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:50.602929 18191 tablet_bootstrap.cc:654] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:50.603138 18190 log.cc:824] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:50.603500 18191 log.cc:824] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:50.603838 18194 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Bootstrap starting.
I20260927 21:01:50.604003 18190 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: No bootstrap required, opened a new log
I20260927 21:01:50.604085 18190 ts_tablet_manager.cc:1397] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:01:50.604454 18194 tablet_bootstrap.cc:654] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:50.604548 18191 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: No bootstrap required, opened a new log
I20260927 21:01:50.604619 18191 ts_tablet_manager.cc:1397] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:01:50.605060 18194 log.cc:824] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:50.605827 18194 tablet_bootstrap.cc:492] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: No bootstrap required, opened a new log
I20260927 21:01:50.605895 18194 ts_tablet_manager.cc:1397] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:01:50.605930 18190 raft_consensus.cc:348] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.606129 18190 raft_consensus.cc:374] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:50.606173 18190 raft_consensus.cc:729] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53dd194cc1634e219d0910642ab628da, State: Initialized, Role: FOLLOWER
I20260927 21:01:50.606292 18190 consensus_queue.cc:260] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.606563 18190 ts_tablet_manager.cc:1428] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:01:50.606688 18042 heartbeater.cc:503] Master 127.16.190.254:38041 was elected leader, sending a full tablet report...
I20260927 21:01:50.606701 18191 raft_consensus.cc:348] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.606887 18191 raft_consensus.cc:374] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:50.606930 18191 raft_consensus.cc:729] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38971215c9104990b2cda57f9fe56967, State: Initialized, Role: FOLLOWER
I20260927 21:01:50.607060 18191 consensus_queue.cc:260] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.607343 18191 ts_tablet_manager.cc:1428] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:50.607533 18176 heartbeater.cc:503] Master 127.16.190.254:38041 was elected leader, sending a full tablet report...
I20260927 21:01:50.608063 18194 raft_consensus.cc:348] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.608220 18194 raft_consensus.cc:374] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:50.608251 18194 raft_consensus.cc:729] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10a3f185aa8471fa1fc050d202f55a6, State: Initialized, Role: FOLLOWER
I20260927 21:01:50.608361 18194 consensus_queue.cc:260] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.608587 18194 ts_tablet_manager.cc:1428] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:50.608633 17908 heartbeater.cc:503] Master 127.16.190.254:38041 was elected leader, sending a full tablet report...
W20260927 21:01:50.657459 18043 tablet.cc:2369] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:01:50.749089 17909 tablet.cc:2369] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:01:50.816886 18177 tablet.cc:2369] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:01:50.848418 18198 raft_consensus.cc:482] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:50.848551 18198 raft_consensus.cc:504] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.849042 18198 leader_election.cc:288] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53dd194cc1634e219d0910642ab628da (127.16.190.194:38265), 38971215c9104990b2cda57f9fe56967 (127.16.190.195:44499)
I20260927 21:01:50.853509 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.849378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:46225 (local address 127.16.190.195:44499)
0927 21:01:50.849421 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.849422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.849612 (+ 190us) server_negotiation.cc:407] Connection header received
0927 21:01:50.849639 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.849640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.849651 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.849729 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:50.849957 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.850753 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.852402 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.853061 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.853138 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.853148 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.853387 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.853437 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:01:50.853509 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.849335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:59257 (local address 127.16.190.194:38265)
0927 21:01:50.849408 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.849410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.849509 (+ 99us) server_negotiation.cc:407] Connection header received
0927 21:01:50.849602 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.849603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.849614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.849676 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:50.849959 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.850716 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.852402 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.853034 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.853140 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.853148 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.853386 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.853437 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:01:50.853506 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.849281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:44499 (local address 127.16.190.193:46225)
0927 21:01:50.849508 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.849518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.849569 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.849742 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.849742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.849744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:50.849816 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.849818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.850835 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.850836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.852281 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.852282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.853161 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.853162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.853257 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.853260 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.853331 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.853384 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":104,"thread_start_us":61,"threads_started":1}
I20260927 21:01:50.853509 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.849264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:38265 (local address 127.16.190.193:59257)
0927 21:01:50.849402 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:50.849413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:50.849510 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:50.849693 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:50.849694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:50.849695 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:50.849813 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.849816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.850835 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.850836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.852278 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:50.852282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.853162 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.853163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:50.853252 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.853257 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:01:50.853325 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:01:50.853384 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":85,"thread_start_us":62,"threads_started":1}
I20260927 21:01:50.854089 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066982050774232b40c7abc6ff20e9b" candidate_uuid: "e10a3f185aa8471fa1fc050d202f55a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38971215c9104990b2cda57f9fe56967" is_pre_election: true
I20260927 21:01:50.854154 17996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066982050774232b40c7abc6ff20e9b" candidate_uuid: "e10a3f185aa8471fa1fc050d202f55a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53dd194cc1634e219d0910642ab628da" is_pre_election: true
I20260927 21:01:50.854280 18130 raft_consensus.cc:2454] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e10a3f185aa8471fa1fc050d202f55a6 in term 0.
I20260927 21:01:50.854316 17996 raft_consensus.cc:2454] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e10a3f185aa8471fa1fc050d202f55a6 in term 0.
I20260927 21:01:50.854559 17796 leader_election.cc:302] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38971215c9104990b2cda57f9fe56967, e10a3f185aa8471fa1fc050d202f55a6; no voters:
I20260927 21:01:50.854758 18198 raft_consensus.cc:2781] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:50.854826 18198 raft_consensus.cc:482] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:50.854844 18198 raft_consensus.cc:3037] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:50.855609 18198 raft_consensus.cc:504] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.855763 18198 leader_election.cc:288] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [CANDIDATE]: Term 1 election: Requested vote from peers 53dd194cc1634e219d0910642ab628da (127.16.190.194:38265), 38971215c9104990b2cda57f9fe56967 (127.16.190.195:44499)
I20260927 21:01:50.855950 18130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066982050774232b40c7abc6ff20e9b" candidate_uuid: "e10a3f185aa8471fa1fc050d202f55a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38971215c9104990b2cda57f9fe56967"
I20260927 21:01:50.855937 17996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066982050774232b40c7abc6ff20e9b" candidate_uuid: "e10a3f185aa8471fa1fc050d202f55a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53dd194cc1634e219d0910642ab628da"
I20260927 21:01:50.856025 18130 raft_consensus.cc:3037] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:50.856026 17996 raft_consensus.cc:3037] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:50.856962 18130 raft_consensus.cc:2454] T 3066982050774232b40c7abc6ff20e9b P 38971215c9104990b2cda57f9fe56967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e10a3f185aa8471fa1fc050d202f55a6 in term 1.
I20260927 21:01:50.856962 17996 raft_consensus.cc:2454] T 3066982050774232b40c7abc6ff20e9b P 53dd194cc1634e219d0910642ab628da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e10a3f185aa8471fa1fc050d202f55a6 in term 1.
I20260927 21:01:50.857194 17795 leader_election.cc:302] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53dd194cc1634e219d0910642ab628da, e10a3f185aa8471fa1fc050d202f55a6; no voters:
I20260927 21:01:50.857312 18198 raft_consensus.cc:2781] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:50.857481 18198 raft_consensus.cc:686] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [term 1 LEADER]: Becoming Leader. State: Replica: e10a3f185aa8471fa1fc050d202f55a6, State: Running, Role: LEADER
I20260927 21:01:50.857612 18198 consensus_queue.cc:237] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } }
I20260927 21:01:50.858629 17717 catalog_manager.cc:5495] T 3066982050774232b40c7abc6ff20e9b P e10a3f185aa8471fa1fc050d202f55a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e10a3f185aa8471fa1fc050d202f55a6 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "e10a3f185aa8471fa1fc050d202f55a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10a3f185aa8471fa1fc050d202f55a6" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53dd194cc1634e219d0910642ab628da" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 38265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38971215c9104990b2cda57f9fe56967" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 44499 } health_report { overall_health: UNKNOWN } } }
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:50.888475 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.879589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.16.190.193:42169)
0927 21:01:50.879668 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.879671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.879820 (+ 149us) server_negotiation.cc:407] Connection header received
0927 21:01:50.879850 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.879851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.879862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.880002 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.880649 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.881277 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.882026 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.882258 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.882322 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.885342 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.885355 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.885355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.885363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.887312 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.887687 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.887690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.887690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.887749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.887927 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.887927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.887927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.888141 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.888285 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.888317 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.888383 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:50.911557 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.904073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.16.190.193:42169)
0927 21:01:50.904180 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.904182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.904222 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:01:50.904298 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.904298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.904308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.904360 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.904976 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.905461 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.906179 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.906388 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.906466 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.908842 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.908844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.908845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.908854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.910234 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.910625 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.910626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.910627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.910715 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.910929 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.910930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.910930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.911044 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.911153 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.911358 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.911450 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:01:50 dist-test-slave-6c7s krb5kdc[17672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:50.934309 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:50.926537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.16.190.193:42169)
0927 21:01:50.926607 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:50.926610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:50.926706 (+ 96us) server_negotiation.cc:407] Connection header received
0927 21:01:50.926795 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:50.926796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:50.926807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:50.926880 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:50.927441 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.927918 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.928659 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:50.928935 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:50.929020 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:50.931421 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:50.931423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:50.931423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:50.931430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:50.933171 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.933517 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.933519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.933519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.933571 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:50.933808 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:50.933809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:50.933809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:50.933913 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:50.934038 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:50.934122 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:01:50.934205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 21:01:50.934820 17822 tablet_service.cc:2908] Rejecting scan request for tablet 3066982050774232b40c7abc6ff20e9b: Uninitialized: clean time has not yet been initialized
I20260927 21:01:50.935734 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17779
I20260927 21:01:50.940765 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17913
I20260927 21:01:50.945873 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18047
I20260927 21:01:50.950371 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 17685
2026-09-27T21:01:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2173 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:01:51 dist-test-slave-6c7s krb5kdc[18215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:01:51 dist-test-slave-6c7s krb5kdc[18215](info): set up 2 sockets
Sep 27 21:01:51 dist-test-slave-6c7s krb5kdc[18215](info): commencing operation
krb5kdc: starting...
W20260927 21:01:51.989104 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:01:52.123377 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36155
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:52.205499 18228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:52.205709 18228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:52.205732 18228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:52.206925 18228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:01:52.206971 18228 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:01:52.206985 18228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:52.206998 18228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:52.207011 18228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:01:52.207065 18228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:01:52.208541 18228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:52.208827 18228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:52.209069 18228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:52.214171 18234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.214222 18233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.214286 18228 server_base.cc:1034] running on GCE node
W20260927 21:01:52.214195 18236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.214598 18228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:52.214906 18228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:52.216058 18228 hybrid_clock.cc:648] HybridClock initialized: now 1790542912216048 us; error 40 us; skew 500 ppm
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:52.226261 18228 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:01:52.226758 18228 webserver.cc:460] Webserver started at http://127.16.190.254:36905/ using document root <none> and password file <none>
I20260927 21:01:52.226958 18228 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:52.226998 18228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:52.227092 18228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:52.228140 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "9206e9b6fc06447b81de843ab4c74155"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "42801e8d783f3d0bff75244dd2d00f8e"
server_key_iv: "2718906efb1d13e3fbd94ccc78249a1a"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.228478 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "9206e9b6fc06447b81de843ab4c74155"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "42801e8d783f3d0bff75244dd2d00f8e"
server_key_iv: "2718906efb1d13e3fbd94ccc78249a1a"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.229904 18228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:01:52.230638 18243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.230866 18228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:01:52.230959 18228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "9206e9b6fc06447b81de843ab4c74155"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "42801e8d783f3d0bff75244dd2d00f8e"
server_key_iv: "2718906efb1d13e3fbd94ccc78249a1a"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.231035 18228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:52.250193 18228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:52.252563 18228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:52.252733 18228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:52.257682 18228 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36155
I20260927 21:01:52.257735 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36155 every 8 connection(s)
I20260927 21:01:52.258071 18228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:01:52.258669 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:52.258948 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18228
I20260927 21:01:52.259053 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:01:52.259202 17147 external_mini_cluster.cc:1428] Setting key 68aa34a752151721d55f0e67f8fa25a4
I20260927 21:01:52.262269 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Bootstrap starting.
I20260927 21:01:52.262953 18306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:52.263250 18306 log.cc:824] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:52.263927 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: No bootstrap required, opened a new log
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:52.266403 18306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } }
I20260927 21:01:52.266548 18306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:52.266570 18306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9206e9b6fc06447b81de843ab4c74155, State: Initialized, Role: FOLLOWER
I20260927 21:01:52.266719 18306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } }
I20260927 21:01:52.266782 18306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:01:52.266803 18306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:01:52.266867 18306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:52.267578 18306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } }
I20260927 21:01:52.267717 18306 leader_election.cc:302] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9206e9b6fc06447b81de843ab4c74155; no voters:
I20260927 21:01:52.267891 18306 leader_election.cc:288] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:01:52.267921 18311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:52.268033 18311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [term 1 LEADER]: Becoming Leader. State: Replica: 9206e9b6fc06447b81de843ab4c74155, State: Running, Role: LEADER
I20260927 21:01:52.268113 18311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } }
I20260927 21:01:52.268186 18306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:01:52.268453 18312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9206e9b6fc06447b81de843ab4c74155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } } }
I20260927 21:01:52.268580 18312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:52.268435 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9206e9b6fc06447b81de843ab4c74155. Latest consensus state: current_term: 1 leader_uuid: "9206e9b6fc06447b81de843ab4c74155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9206e9b6fc06447b81de843ab4c74155" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36155 } } }
I20260927 21:01:52.268901 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155 [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:52.269133 18318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:01:52.269695 18318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:01:52.270901 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.259749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.16.190.254:36155)
0927 21:01:52.259942 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.259952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.259988 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:01:52.260477 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.260486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.260551 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.260718 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.261371 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.262748 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.263562 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.263963 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.264082 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.266729 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.266734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.266734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.266748 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.268968 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.269433 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.269436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.269436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.269557 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.269818 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.269819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.269819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.270149 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.270286 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.270564 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.270706 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260927 21:01:52.271647 18318 catalog_manager.cc:1347] Generated new cluster ID: d178613c88ff4ed8a918a22658f7a5f9
I20260927 21:01:52.271698 18318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:01:52.282997 18318 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:01:52.283741 18318 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:01:52.291028 18318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Generated new TSK 0
I20260927 21:01:52.291254 18318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:01:52.320706 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36155
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:52.406324 18332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:52.406531 18332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:52.406562 18332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:52.407647 18332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:52.407676 18332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:52.407696 18332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:52.407714 18332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:01:52.409157 18332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:52.409340 18332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:52.409513 18332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:52.414740 18340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.414741 18338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.414742 18337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.415130 18332 server_base.cc:1034] running on GCE node
I20260927 21:01:52.415380 18332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:52.415649 18332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:52.416854 18332 hybrid_clock.cc:648] HybridClock initialized: now 1790542912416850 us; error 43 us; skew 500 ppm
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:52.419571 18332 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:01:52.420022 18332 webserver.cc:460] Webserver started at http://127.16.190.193:39803/ using document root <none> and password file <none>
I20260927 21:01:52.420256 18332 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:52.420325 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:52.420477 18332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:52.421643 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "6d95e32416cc4bfb893efca4cdc7e218"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "8b9e00a2090d60535a0fe61de979a4bd"
server_key_iv: "d675bf56834cc1a9d01a9b99228df233"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.422015 18332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "6d95e32416cc4bfb893efca4cdc7e218"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "8b9e00a2090d60535a0fe61de979a4bd"
server_key_iv: "d675bf56834cc1a9d01a9b99228df233"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.423349 18332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:01:52.424091 18347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.424266 18332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.424371 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "6d95e32416cc4bfb893efca4cdc7e218"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "8b9e00a2090d60535a0fe61de979a4bd"
server_key_iv: "d675bf56834cc1a9d01a9b99228df233"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.424446 18332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:52.443862 18332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:52.446416 18332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:52.446583 18332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:52.447197 18332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:52.447256 18332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.447291 18332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:52.447314 18332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.454715 18332 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42529
I20260927 21:01:52.454816 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42529 every 8 connection(s)
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:52.455127 18332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:01:52.455786 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18332
I20260927 21:01:52.455891 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:01:52.456087 17147 external_mini_cluster.cc:1428] Setting key a1b42a8823274a797025cc37c3538e97
I20260927 21:01:52.458719 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.447719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:35359 (local address 127.16.190.254:36155)
0927 21:01:52.447794 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.447796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.447999 (+ 203us) server_negotiation.cc:407] Connection header received
0927 21:01:52.448759 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.448761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.448771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.448879 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.449809 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.450671 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.452343 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.452692 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.452750 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.455460 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.455462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.455463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.455473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.457136 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.457659 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.457659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.457660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.457734 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.458009 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.458009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.458010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.458158 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.458235 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.458513 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.458599 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:01:52.458899 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.447670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.193:35359)
0927 21:01:52.447851 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.447902 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.448630 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.448970 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.448976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.449384 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.449713 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.449719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.450756 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.450757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.452242 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.452246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.452783 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.452784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.452972 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.453651 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.453660 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.455339 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.457221 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.457228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.457230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.457580 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.457799 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.457800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.457800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.457946 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.458229 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.458236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.458379 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.458492 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.458656 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260927 21:01:52.459592 18463 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36155
I20260927 21:01:52.459683 18463 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:52.459879 18463 heartbeater.cc:511] Master 127.16.190.254:36155 requested a full tablet report, sending...
I20260927 21:01:52.460429 18259 ts_manager.cc:194] Registered new tserver with Master: 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529)
I20260927 21:01:52.461287 18259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:35359
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:01:52.466522 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.460770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:60673 (local address 127.16.190.254:36155)
0927 21:01:52.460826 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.460828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.460862 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:01:52.460925 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.460926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.460935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.460986 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.461402 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.462013 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.462893 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.463099 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.463182 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.464100 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.464102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.464102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.464109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.465500 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.465907 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.465908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.465909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.465961 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.466135 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.466135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.466135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.466237 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.466287 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.466398 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.466459 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:01:52.466527 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.460652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.193:60673)
0927 21:01:52.460782 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.460795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.460877 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.461002 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.461003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.461243 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.461311 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.461313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.462114 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.462115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.462803 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.462805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.463179 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.463180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.463256 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.463716 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.463720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.463997 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.465594 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.465595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.465595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.465824 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.466025 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.466026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.466026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.466080 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.466292 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.466293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.466331 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.466381 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.466462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:01:52.483590 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36155
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:52.568341 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:52.568632 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:52.568660 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:52.569976 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:52.570025 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:52.570048 18472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:52.570066 18472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:01:52.571589 18472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:52.571859 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:52.572108 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:52.577382 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.577423 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.577404 18480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.577527 18472 server_base.cc:1034] running on GCE node
I20260927 21:01:52.577831 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:52.578084 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:52.579253 18472 hybrid_clock.cc:648] HybridClock initialized: now 1790542912579232 us; error 41 us; skew 500 ppm
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:52.581852 18472 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:01:52.582312 18472 webserver.cc:460] Webserver started at http://127.16.190.194:45013/ using document root <none> and password file <none>
I20260927 21:01:52.582551 18472 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:52.582614 18472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:52.582762 18472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:52.583832 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "ad065bc1d75941cf890dee87c128da75"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "cc35ea78c39f32e1f4c16620fd664b2a"
server_key_iv: "a363524e316af3716b37e0da89938767"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.584192 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "ad065bc1d75941cf890dee87c128da75"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "cc35ea78c39f32e1f4c16620fd664b2a"
server_key_iv: "a363524e316af3716b37e0da89938767"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.585602 18472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:01:52.586413 18487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.586665 18472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:52.586766 18472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "ad065bc1d75941cf890dee87c128da75"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "cc35ea78c39f32e1f4c16620fd664b2a"
server_key_iv: "a363524e316af3716b37e0da89938767"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.586838 18472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:52.608863 18472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:52.611469 18472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:52.611694 18472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:52.612258 18472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:52.612329 18472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.612365 18472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:52.612396 18472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.619680 18472 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44595
I20260927 21:01:52.619766 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44595 every 8 connection(s)
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:52.620052 18472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:01:52.623441 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.612712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:37991 (local address 127.16.190.254:36155)
0927 21:01:52.612768 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.612770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.613063 (+ 293us) server_negotiation.cc:407] Connection header received
0927 21:01:52.613836 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.613837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.613853 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.613926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.614864 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.615722 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.617442 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.617678 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.617738 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.620571 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.620572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.620573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.620581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.622032 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.622459 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.622461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.622462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.622521 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.622783 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.622784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.622784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.622907 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.622983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.623269 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.623353 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:01:52.623638 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.612696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.194:37991)
0927 21:01:52.612894 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.612951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.613691 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.614012 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.614017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.614442 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.614765 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.614772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.615827 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.615828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.617346 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.617350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.617765 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.617767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.617951 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.618559 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.618571 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.620486 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.622134 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.622140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.622143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.622370 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.622598 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.622599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.622599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.622709 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.622975 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.622982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.623126 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.623235 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.623393 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260927 21:01:52.624136 18603 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36155
I20260927 21:01:52.624229 18603 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:52.624420 18603 heartbeater.cc:511] Master 127.16.190.254:36155 requested a full tablet report, sending...
I20260927 21:01:52.624778 18259 ts_manager.cc:194] Registered new tserver with Master: ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595)
I20260927 21:01:52.625289 18259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:37991
I20260927 21:01:52.629065 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18472
I20260927 21:01:52.629163 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:01:52.629367 17147 external_mini_cluster.cc:1428] Setting key e61fc052e9b518cbdeeb4c0ad74c6100
I20260927 21:01:52.630837 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.625251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:45821 (local address 127.16.190.254:36155)
0927 21:01:52.625422 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.625424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.625445 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:01:52.625465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.625466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.625474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.625521 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.625907 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.626364 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.627019 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.627298 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.627356 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.628114 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.628115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.628115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.628120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.629689 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.630028 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.630030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.630030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.630089 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.630289 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.630290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.630290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.630437 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.630516 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.630671 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.630746 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260927 21:01:52.630853 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.625181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.194:45821)
0927 21:01:52.625316 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.625326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.625398 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.625523 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.625524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.625760 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.625828 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.625830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.626456 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.626456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.626949 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.626951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.627368 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.627368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.627424 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.627770 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.627774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.628028 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.629769 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.629771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.629771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.629951 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.630152 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.630153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.630153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.630211 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.630507 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.630508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.630557 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.630646 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.630748 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:01:52.656949 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36155
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:52.738339 18612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:52.738574 18612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:52.738615 18612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:52.739760 18612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:52.739825 18612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:52.739861 18612 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:52.739918 18612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:01:52.741415 18612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:36155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:52.741670 18612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:52.741889 18612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:52.747092 18618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:52.747102 18620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.747134 18612 server_base.cc:1034] running on GCE node
W20260927 21:01:52.747121 18617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:52.747633 18612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:52.747856 18612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:52.749027 18612 hybrid_clock.cc:648] HybridClock initialized: now 1790542912749012 us; error 45 us; skew 500 ppm
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:52.751619 18612 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:01:52.752097 18612 webserver.cc:460] Webserver started at http://127.16.190.195:39725/ using document root <none> and password file <none>
I20260927 21:01:52.752331 18612 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:52.752377 18612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:52.752502 18612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:52.753624 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "186739a517ba4a9ea54760d6b85343ef"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "e1922cec4e8cc60469f1475cd6342527"
server_key_iv: "5a4789a7f2a06daf5a2bd46362499e00"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.754000 18612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "186739a517ba4a9ea54760d6b85343ef"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "e1922cec4e8cc60469f1475cd6342527"
server_key_iv: "5a4789a7f2a06daf5a2bd46362499e00"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.755403 18612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:01:52.756178 18627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.756373 18612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:52.756458 18612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "186739a517ba4a9ea54760d6b85343ef"
format_stamp: "Formatted at 2026-09-27 21:01:52 on dist-test-slave-6c7s"
server_key: "e1922cec4e8cc60469f1475cd6342527"
server_key_iv: "5a4789a7f2a06daf5a2bd46362499e00"
server_key_version: "encryptionkey@0"
I20260927 21:01:52.756515 18612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:52.765201 18612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:52.767557 18612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:52.767676 18612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:52.768244 18612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:52.768288 18612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.768313 18612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:52.768326 18612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:52.775288 18612 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:43681
I20260927 21:01:52.775388 18742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:43681 every 8 connection(s)
I20260927 21:01:52.775727 18612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:52.779518 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.768851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:43645 (local address 127.16.190.254:36155)
0927 21:01:52.768900 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.768903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.769240 (+ 337us) server_negotiation.cc:407] Connection header received
0927 21:01:52.769990 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.769993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.770008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.770041 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.771028 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.771842 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.773591 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.773830 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.773923 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.776697 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.776698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.776699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.776707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.778082 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.778493 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.778494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.778495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.778562 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.778838 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.778839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.778839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.778961 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.779033 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.779341 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.779427 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:01:52.779706 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.768862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.195:43645)
0927 21:01:52.769064 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.769134 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.769810 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.770258 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.770264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.770651 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.770938 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.770944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.771971 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.771973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.773483 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.773487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.773928 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.773929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.774082 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.774771 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.774781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.776592 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.778174 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.778179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.778182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.778412 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.778647 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.778648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.778648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.778764 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.779034 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.779041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.779196 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.779301 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.779464 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260927 21:01:52.780156 18743 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36155
I20260927 21:01:52.780267 18743 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:52.780459 18743 heartbeater.cc:511] Master 127.16.190.254:36155 requested a full tablet report, sending...
I20260927 21:01:52.780939 18259 ts_manager.cc:194] Registered new tserver with Master: 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681)
I20260927 21:01:52.781390 18259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:43645
I20260927 21:01:52.781855 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18612
I20260927 21:01:52.781920 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:01:52.782091 17147 external_mini_cluster.cc:1428] Setting key cbb806c664a6ec2e43db6d76fc1e0f0d
I20260927 21:01:52.782991 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:01:52.787686 18749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.781209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.16.190.195:52537)
0927 21:01:52.781336 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.781346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.781398 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.781613 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.781614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.781816 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:52.781880 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.781882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.782808 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.782809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.783435 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.783437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.783914 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.783915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.783981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.784497 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:52.784502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:52.784794 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:52.786586 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.786587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.786588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.786773 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.786997 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:52.786998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:52.786998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:52.787072 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:52.787329 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:52.787330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:52.787378 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.787468 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.787587 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260927 21:01:52.787683 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.781276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:52537 (local address 127.16.190.254:36155)
0927 21:01:52.781485 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.781487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.781494 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:01:52.781529 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.781530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.781540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.781607 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.781968 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.782726 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.783518 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.783819 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.783898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.784925 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.784927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.784927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.784934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.786437 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.786860 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.786861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.786862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.786926 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.787138 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.787139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.787140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.787259 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.787338 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.787506 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.787584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:01:52 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:52.807222 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.799804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.16.190.254:36155)
0927 21:01:52.799890 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.799892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.799902 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:01:52.799929 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.799930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.799940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.800067 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:52.800773 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.801350 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.802121 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.802319 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.802408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.804794 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:52.804796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:52.804796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:52.804805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:52.806075 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.806422 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.806424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.806424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.806486 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:52.806688 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:52.806689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:52.806689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:52.806804 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:52.806910 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.807034 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.807113 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:01:52.808754 18259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:52.810400 18259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:52.817462 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.813037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.16.190.193:42529)
0927 21:01:52.813579 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.813583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.813593 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:01:52.813632 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.813635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.813650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.813726 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:52.813969 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.815116 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.816404 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817029 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817124 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817189 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.817321 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.817361 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":96,"threads_started":1}
I20260927 21:01:52.817555 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.812980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44595 (local address 127.0.0.1:58148)
0927 21:01:52.813421 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.813429 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.813463 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.813759 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.813763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.813767 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:52.813851 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.813854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.815234 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.815235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.816191 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.816194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817271 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.817341 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817346 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.817398 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.817446 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":11,"thread_start_us":131,"threads_started":1}
I20260927 21:01:52.817670 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.813459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.16.190.195:43681)
0927 21:01:52.813643 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.813647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.813656 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:52.813700 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.813702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.813718 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.813793 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:52.813972 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.814766 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.816334 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817102 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817179 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817232 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.817491 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.817555 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260927 21:01:52.817816 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.812980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43681 (local address 127.0.0.1:35424)
0927 21:01:52.813209 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.813245 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.813317 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.813852 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.813853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.813855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:52.813914 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.813915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.814875 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.814876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.816252 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.816255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817273 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.817349 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817351 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.817398 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.817446 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":60,"thread_start_us":130,"threads_started":1}
I20260927 21:01:52.817834 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.813271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.16.190.194:44595)
0927 21:01:52.813446 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:52.813452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:52.813471 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:01:52.813527 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:52.813530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:52.813547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:52.813647 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:52.813945 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.815138 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.816288 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817102 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817186 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817233 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:52.817664 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 21:01:52.817740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260927 21:01:52.817456 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:52.812934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42529 (local address 127.0.0.1:48304)
0927 21:01:52.813198 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:52.813245 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:52.813316 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:52.813800 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:52.813801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:52.813803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:52.813866 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.813867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.815225 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.815226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.816308 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:52.816310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:52.817122 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:52.817123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:52.817208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:52.817223 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:01:52.817298 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:01:52.817358 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":51,"threads_started":1}
I20260927 21:01:52.818643 18677 tablet_service.cc:1467] Processing CreateTablet for tablet f75379b5b53c47a8888e95e01bf87b3d (DEFAULT_TABLE table=test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:52.818701 18537 tablet_service.cc:1467] Processing CreateTablet for tablet f75379b5b53c47a8888e95e01bf87b3d (DEFAULT_TABLE table=test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:52.819031 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75379b5b53c47a8888e95e01bf87b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:52.819031 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75379b5b53c47a8888e95e01bf87b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:52.819177 18397 tablet_service.cc:1467] Processing CreateTablet for tablet f75379b5b53c47a8888e95e01bf87b3d (DEFAULT_TABLE table=test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:52.819526 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75379b5b53c47a8888e95e01bf87b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:52.822532 18766 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Bootstrap starting.
I20260927 21:01:52.822768 18767 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Bootstrap starting.
I20260927 21:01:52.822817 18768 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Bootstrap starting.
I20260927 21:01:52.823382 18768 tablet_bootstrap.cc:654] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:52.823622 18767 tablet_bootstrap.cc:654] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:52.823728 18768 log.cc:824] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:52.823963 18767 log.cc:824] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:52.824445 18768 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: No bootstrap required, opened a new log
I20260927 21:01:52.824517 18768 ts_tablet_manager.cc:1397] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:52.824891 18767 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: No bootstrap required, opened a new log
I20260927 21:01:52.824967 18767 ts_tablet_manager.cc:1397] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:01:52.824992 18766 tablet_bootstrap.cc:654] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:52.825315 18766 log.cc:824] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:52.826088 18766 tablet_bootstrap.cc:492] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: No bootstrap required, opened a new log
I20260927 21:01:52.826156 18766 ts_tablet_manager.cc:1397] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 21:01:52.826458 18768 raft_consensus.cc:348] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.826663 18768 raft_consensus.cc:374] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:52.826709 18768 raft_consensus.cc:729] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad065bc1d75941cf890dee87c128da75, State: Initialized, Role: FOLLOWER
I20260927 21:01:52.826822 18768 consensus_queue.cc:260] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.827111 18603 heartbeater.cc:503] Master 127.16.190.254:36155 was elected leader, sending a full tablet report...
I20260927 21:01:52.827145 18767 raft_consensus.cc:348] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.827307 18767 raft_consensus.cc:374] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:52.827363 18767 raft_consensus.cc:729] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186739a517ba4a9ea54760d6b85343ef, State: Initialized, Role: FOLLOWER
I20260927 21:01:52.827097 18768 ts_tablet_manager.cc:1428] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:52.827531 18767 consensus_queue.cc:260] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.827812 18767 ts_tablet_manager.cc:1428] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:52.827862 18743 heartbeater.cc:503] Master 127.16.190.254:36155 was elected leader, sending a full tablet report...
I20260927 21:01:52.828374 18766 raft_consensus.cc:348] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.828537 18766 raft_consensus.cc:374] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:52.828591 18766 raft_consensus.cc:729] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d95e32416cc4bfb893efca4cdc7e218, State: Initialized, Role: FOLLOWER
I20260927 21:01:52.828716 18766 consensus_queue.cc:260] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:52.829011 18766 ts_tablet_manager.cc:1428] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:52.829146 18463 heartbeater.cc:503] Master 127.16.190.254:36155 was elected leader, sending a full tablet report...
W20260927 21:01:52.870770 18604 tablet.cc:2369] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:01:52.955920 18464 tablet.cc:2369] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:01:53.026278 18744 tablet.cc:2369] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:01:53.154822 18774 raft_consensus.cc:482] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:53.155009 18774 raft_consensus.cc:504] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:53.155418 18774 leader_election.cc:288] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681), ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595)
I20260927 21:01:53.158452 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.155684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:57493 (local address 127.16.190.194:44595)
0927 21:01:53.155733 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.155736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.155818 (+ 82us) server_negotiation.cc:407] Connection header received
0927 21:01:53.155896 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.155898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.155907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.155984 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.156199 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.156655 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.157719 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.158087 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.158174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.158185 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.158330 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.158374 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:01:53.158458 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.155764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:39935 (local address 127.16.190.195:43681)
0927 21:01:53.155815 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.155817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.155829 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:01:53.155882 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.155883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.155899 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.155943 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.156121 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.156617 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.157729 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.158087 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.158196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.158205 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.158356 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.158393 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:01:53.158478 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.155606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44595 (local address 127.16.190.193:57493)
0927 21:01:53.155741 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.155752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.155821 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.156002 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.156003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.156005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.156100 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.156102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.156784 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.156785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.157613 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.157615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.158174 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.158176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.158252 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.158256 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.158309 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.158370 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260927 21:01:53.158483 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.155624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43681 (local address 127.16.190.193:39935)
0927 21:01:53.155748 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.155759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.155815 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.155972 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.155973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.155975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.156050 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.156052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.156743 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.156744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.157610 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.157613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.158198 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.158200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.158265 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.158268 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.158343 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.158384 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":14,"thread_start_us":59,"threads_started":1}
I20260927 21:01:53.158815 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75379b5b53c47a8888e95e01bf87b3d" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186739a517ba4a9ea54760d6b85343ef" is_pre_election: true
I20260927 21:01:53.158887 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75379b5b53c47a8888e95e01bf87b3d" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75" is_pre_election: true
I20260927 21:01:53.158979 18697 raft_consensus.cc:2454] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 0.
I20260927 21:01:53.159072 18557 raft_consensus.cc:2454] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 0.
I20260927 21:01:53.159238 18351 leader_election.cc:302] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186739a517ba4a9ea54760d6b85343ef, 6d95e32416cc4bfb893efca4cdc7e218; no voters:
I20260927 21:01:53.159368 18774 raft_consensus.cc:2781] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:53.159435 18774 raft_consensus.cc:482] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:53.159453 18774 raft_consensus.cc:3037] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.160431 18774 raft_consensus.cc:504] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:53.160594 18774 leader_election.cc:288] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 election: Requested vote from peers 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681), ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595)
I20260927 21:01:53.160748 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75379b5b53c47a8888e95e01bf87b3d" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186739a517ba4a9ea54760d6b85343ef"
I20260927 21:01:53.160799 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75379b5b53c47a8888e95e01bf87b3d" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75"
I20260927 21:01:53.160866 18697 raft_consensus.cc:3037] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.160925 18557 raft_consensus.cc:3037] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.161633 18697 raft_consensus.cc:2454] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 1.
I20260927 21:01:53.161650 18557 raft_consensus.cc:2454] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 1.
I20260927 21:01:53.161828 18351 leader_election.cc:302] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186739a517ba4a9ea54760d6b85343ef, 6d95e32416cc4bfb893efca4cdc7e218; no voters:
I20260927 21:01:53.161958 18774 raft_consensus.cc:2781] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:53.162102 18774 raft_consensus.cc:686] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 LEADER]: Becoming Leader. State: Replica: 6d95e32416cc4bfb893efca4cdc7e218, State: Running, Role: LEADER
I20260927 21:01:53.162221 18774 consensus_queue.cc:237] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } }
I20260927 21:01:53.163074 18259 catalog_manager.cc:5495] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } health_report { overall_health: UNKNOWN } } }
I20260927 21:01:53.171242 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.168507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.16.190.193:42529)
0927 21:01:53.168566 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.168568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.168674 (+ 106us) server_negotiation.cc:407] Connection header received
0927 21:01:53.168736 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.168737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.168755 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.168882 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:53.169042 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.169622 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.170438 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.170651 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.170710 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.170893 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.170987 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.171054 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.171164 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.171184 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:01:53.173071 18557 raft_consensus.cc:1261] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Refusing update from remote peer 6d95e32416cc4bfb893efca4cdc7e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.173071 18697 raft_consensus.cc:1261] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Refusing update from remote peer 6d95e32416cc4bfb893efca4cdc7e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.173424 18777 consensus_queue.cc:1035] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.173612 18774 consensus_queue.cc:1035] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.176152 18781 mvcc.cc:205] Tried to move back new op lower bound from 7334063772354068480 to 7334063772313128960. Current Snapshot: MvccSnapshot[applied={T|T < 7334063772354068480}]
I20260927 21:01:53.176733 18782 mvcc.cc:205] Tried to move back new op lower bound from 7334063772354068480 to 7334063772313128960. Current Snapshot: MvccSnapshot[applied={T|T < 7334063772354068480}]
I20260927 21:01:53.179493 18259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:53.179612 18259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:53.180344 18259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Sending DeleteTablet for 3 replicas of tablet f75379b5b53c47a8888e95e01bf87b3d
I20260927 21:01:53.180660 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet f75379b5b53c47a8888e95e01bf87b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260927 21:01:53.180781 18537 tablet_service.cc:1514] Processing DeleteTablet for tablet f75379b5b53c47a8888e95e01bf87b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58148
I20260927 21:01:53.180894 18677 tablet_service.cc:1514] Processing DeleteTablet for tablet f75379b5b53c47a8888e95e01bf87b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35424
I20260927 21:01:53.180933 18790 tablet_replica.cc:330] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: stopping tablet replica
I20260927 21:01:53.181027 18790 raft_consensus.cc:2229] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:53.181146 18790 raft_consensus.cc:2258] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.181313 18792 tablet_replica.cc:330] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: stopping tablet replica
I20260927 21:01:53.181409 18792 raft_consensus.cc:2229] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:53.181512 18792 raft_consensus.cc:2258] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.181499 18259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:53.181672 18259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:53.181789 18791 tablet_replica.cc:330] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: stopping tablet replica
I20260927 21:01:53.181918 18792 ts_tablet_manager.cc:1905] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.181928 18791 raft_consensus.cc:2229] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:53.182108 18791 raft_consensus.cc:2258] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.182950 18397 tablet_service.cc:1467] Processing CreateTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 (DEFAULT_TABLE table=test-table [id=6f78327e8feb492ebe54ee0173ed059c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:53.183010 18537 tablet_service.cc:1467] Processing CreateTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 (DEFAULT_TABLE table=test-table [id=6f78327e8feb492ebe54ee0173ed059c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:53.183099 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53795f7e77ba4dd2998e500e84705fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.183099 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53795f7e77ba4dd2998e500e84705fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.183557 18792 ts_tablet_manager.cc:1918] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:01:53.183619 18792 log.cc:1196] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/f75379b5b53c47a8888e95e01bf87b3d
I20260927 21:01:53.183866 18792 ts_tablet_manager.cc:1939] T f75379b5b53c47a8888e95e01bf87b3d P 186739a517ba4a9ea54760d6b85343ef: Deleting consensus metadata
I20260927 21:01:53.184276 18247 catalog_manager.cc:4841] TS 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681): tablet f75379b5b53c47a8888e95e01bf87b3d (table test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]) successfully deleted
I20260927 21:01:53.184393 18766 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Bootstrap starting.
I20260927 21:01:53.184412 18677 tablet_service.cc:1467] Processing CreateTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 (DEFAULT_TABLE table=test-table [id=6f78327e8feb492ebe54ee0173ed059c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:53.184504 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53795f7e77ba4dd2998e500e84705fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.184870 18766 tablet_bootstrap.cc:654] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.185585 18766 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: No bootstrap required, opened a new log
I20260927 21:01:53.185668 18766 ts_tablet_manager.cc:1397] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:53.185685 18767 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Bootstrap starting.
I20260927 21:01:53.185722 18768 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Bootstrap starting.
I20260927 21:01:53.185808 18766 raft_consensus.cc:348] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.185877 18766 raft_consensus.cc:374] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.185897 18766 raft_consensus.cc:729] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d95e32416cc4bfb893efca4cdc7e218, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.185943 18766 consensus_queue.cc:260] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.186033 18766 ts_tablet_manager.cc:1428] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:53.186093 18767 tablet_bootstrap.cc:654] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.186093 18768 tablet_bootstrap.cc:654] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.186697 18768 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: No bootstrap required, opened a new log
I20260927 21:01:53.186745 18768 ts_tablet_manager.cc:1397] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:53.186918 18790 ts_tablet_manager.cc:1905] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.186905 18768 raft_consensus.cc:348] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.186990 18768 raft_consensus.cc:374] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.187031 18768 raft_consensus.cc:729] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad065bc1d75941cf890dee87c128da75, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.187078 18768 consensus_queue.cc:260] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.187188 18768 ts_tablet_manager.cc:1428] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:53.188555 18790 ts_tablet_manager.cc:1918] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:01:53.188632 18790 log.cc:1196] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/f75379b5b53c47a8888e95e01bf87b3d
I20260927 21:01:53.188704 18791 ts_tablet_manager.cc:1905] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.188928 18790 ts_tablet_manager.cc:1939] T f75379b5b53c47a8888e95e01bf87b3d P 6d95e32416cc4bfb893efca4cdc7e218: Deleting consensus metadata
I20260927 21:01:53.189462 18245 catalog_manager.cc:4841] TS 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529): tablet f75379b5b53c47a8888e95e01bf87b3d (table test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]) successfully deleted
I20260927 21:01:53.190513 18791 ts_tablet_manager.cc:1918] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:01:53.190567 18791 log.cc:1196] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/f75379b5b53c47a8888e95e01bf87b3d
I20260927 21:01:53.190827 18791 ts_tablet_manager.cc:1939] T f75379b5b53c47a8888e95e01bf87b3d P ad065bc1d75941cf890dee87c128da75: Deleting consensus metadata
I20260927 21:01:53.191334 18767 tablet_bootstrap.cc:492] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: No bootstrap required, opened a new log
I20260927 21:01:53.191354 18246 catalog_manager.cc:4841] TS ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595): tablet f75379b5b53c47a8888e95e01bf87b3d (table test-table [id=b635de51ea874cdcb3de1fb544c9bcb0]) successfully deleted
I20260927 21:01:53.191411 18767 ts_tablet_manager.cc:1397] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 21:01:53.191581 18767 raft_consensus.cc:348] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.191663 18767 raft_consensus.cc:374] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.191684 18767 raft_consensus.cc:729] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186739a517ba4a9ea54760d6b85343ef, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.191726 18767 consensus_queue.cc:260] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.191843 18767 ts_tablet_manager.cc:1428] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:53.218237 18773 raft_consensus.cc:482] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:53.218336 18773 raft_consensus.cc:504] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.218727 18773 leader_election.cc:288] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529), ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595)
I20260927 21:01:53.222311 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.219005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:40727 (local address 127.16.190.193:42529)
0927 21:01:53.219049 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.219051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.219061 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:01:53.219126 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.219127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.219138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.219201 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.219425 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.220183 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.221572 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.221953 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.222040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.222050 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.222211 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.222252 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:53.222308 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.218914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42529 (local address 127.16.190.195:40727)
0927 21:01:53.218979 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.218992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.219045 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.219250 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.219250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.219252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.219336 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.219339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.220267 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.220268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.221445 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.221448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.222036 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.222037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.222121 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.222124 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.222191 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.222232 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260927 21:01:53.222679 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.219094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:45107 (local address 127.16.190.194:44595)
0927 21:01:53.219141 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.219143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.219300 (+ 157us) server_negotiation.cc:407] Connection header received
0927 21:01:53.219338 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.219339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.219349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.219401 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.219569 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.220034 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.221544 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.222126 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.222202 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.222210 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.222564 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.222607 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:01:53.222616 18417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53795f7e77ba4dd2998e500e84705fc8" candidate_uuid: "186739a517ba4a9ea54760d6b85343ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d95e32416cc4bfb893efca4cdc7e218" is_pre_election: true
I20260927 21:01:53.222795 18417 raft_consensus.cc:2454] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186739a517ba4a9ea54760d6b85343ef in term 0.
I20260927 21:01:53.222924 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.218920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44595 (local address 127.16.190.195:45107)
0927 21:01:53.219216 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.219225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.219290 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.219433 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.219434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.219435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.219503 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.219504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.220124 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.220125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.221448 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.221450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.222260 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.222261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.222334 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.222338 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.222801 (+ 463us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.222842 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":162,"thread_start_us":63,"threads_started":1}
I20260927 21:01:53.222991 18629 leader_election.cc:302] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186739a517ba4a9ea54760d6b85343ef, 6d95e32416cc4bfb893efca4cdc7e218; no voters:
I20260927 21:01:53.223137 18773 raft_consensus.cc:2781] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:53.223181 18773 raft_consensus.cc:482] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:53.223207 18773 raft_consensus.cc:3037] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.223357 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53795f7e77ba4dd2998e500e84705fc8" candidate_uuid: "186739a517ba4a9ea54760d6b85343ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75" is_pre_election: true
I20260927 21:01:53.223448 18557 raft_consensus.cc:2454] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 186739a517ba4a9ea54760d6b85343ef in term 0.
I20260927 21:01:53.224069 18773 raft_consensus.cc:504] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.224285 18773 leader_election.cc:288] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [CANDIDATE]: Term 1 election: Requested vote from peers 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529), ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595)
I20260927 21:01:53.224436 18417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53795f7e77ba4dd2998e500e84705fc8" candidate_uuid: "186739a517ba4a9ea54760d6b85343ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d95e32416cc4bfb893efca4cdc7e218"
I20260927 21:01:53.224476 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53795f7e77ba4dd2998e500e84705fc8" candidate_uuid: "186739a517ba4a9ea54760d6b85343ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75"
I20260927 21:01:53.224520 18417 raft_consensus.cc:3037] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.224539 18557 raft_consensus.cc:3037] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.225358 18417 raft_consensus.cc:2454] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186739a517ba4a9ea54760d6b85343ef in term 1.
I20260927 21:01:53.225358 18557 raft_consensus.cc:2454] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 186739a517ba4a9ea54760d6b85343ef in term 1.
I20260927 21:01:53.225543 18630 leader_election.cc:302] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186739a517ba4a9ea54760d6b85343ef, ad065bc1d75941cf890dee87c128da75; no voters:
I20260927 21:01:53.225666 18773 raft_consensus.cc:2781] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:53.225831 18773 raft_consensus.cc:686] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 1 LEADER]: Becoming Leader. State: Replica: 186739a517ba4a9ea54760d6b85343ef, State: Running, Role: LEADER
I20260927 21:01:53.225904 18773 consensus_queue.cc:237] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.226686 18260 catalog_manager.cc:5495] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 186739a517ba4a9ea54760d6b85343ef (127.16.190.195). New cstate: current_term: 1 leader_uuid: "186739a517ba4a9ea54760d6b85343ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } health_report { overall_health: HEALTHY } } }
Sep 27 21:01:53 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:53.269665 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.261621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.16.190.254:36155)
0927 21:01:53.261688 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.261691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.261779 (+ 88us) server_negotiation.cc:407] Connection header received
0927 21:01:53.261816 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.261817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.261828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.261912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:53.262346 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.263136 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.263853 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.264137 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.264218 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.266508 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:53.266510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:53.266510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:53.266518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:53.268200 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.268762 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.268763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.268763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.268848 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.269082 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.269083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.269084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.269283 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:53.269339 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.269517 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.269594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:01:53.269824 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.261509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36155 (local address 127.0.0.1:35620)
0927 21:01:53.261710 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.261719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.261775 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.261922 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.261923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.262205 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:53.262256 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.262258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.263230 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.263231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.263766 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.263767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.264275 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.264277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.264343 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.265002 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:53.265012 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:53.266402 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:53.268304 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:53.268311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:53.268313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:53.268684 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:53.268930 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:53.268931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:53.268932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:53.268997 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:53.269337 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:53.269345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:53.269404 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.269486 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.269722 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":52,"thread_start_us":83,"threads_started":1}
I20260927 21:01:53.272143 18260 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35620:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:01:53.272387 18260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:53.273866 18397 tablet_service.cc:1467] Processing CreateTablet for tablet 4138724fdcff406dac5d1391cac0bf69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb4a595ec8c94336be027a0e18b68900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:53.273934 18537 tablet_service.cc:1467] Processing CreateTablet for tablet 4138724fdcff406dac5d1391cac0bf69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb4a595ec8c94336be027a0e18b68900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:53.273927 18677 tablet_service.cc:1467] Processing CreateTablet for tablet 4138724fdcff406dac5d1391cac0bf69 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb4a595ec8c94336be027a0e18b68900]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:53.274035 18397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4138724fdcff406dac5d1391cac0bf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.274072 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4138724fdcff406dac5d1391cac0bf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.274073 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4138724fdcff406dac5d1391cac0bf69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:53.275548 18766 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: Bootstrap starting.
I20260927 21:01:53.275559 18768 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: Bootstrap starting.
I20260927 21:01:53.275930 18768 tablet_bootstrap.cc:654] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.275969 18766 tablet_bootstrap.cc:654] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.276350 18767 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: Bootstrap starting.
I20260927 21:01:53.276602 18768 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: No bootstrap required, opened a new log
I20260927 21:01:53.276654 18768 ts_tablet_manager.cc:1397] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:53.276799 18767 tablet_bootstrap.cc:654] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:53.276800 18768 raft_consensus.cc:348] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.276916 18768 raft_consensus.cc:374] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.276937 18768 raft_consensus.cc:729] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad065bc1d75941cf890dee87c128da75, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.276975 18768 consensus_queue.cc:260] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.277094 18768 ts_tablet_manager.cc:1428] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:53.277288 18772 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.277395 18772 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.277876 18766 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: No bootstrap required, opened a new log
I20260927 21:01:53.277932 18766 ts_tablet_manager.cc:1397] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:53.278034 18767 tablet_bootstrap.cc:492] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: No bootstrap required, opened a new log
I20260927 21:01:53.278087 18767 ts_tablet_manager.cc:1397] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:53.278083 18766 raft_consensus.cc:348] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.278173 18766 raft_consensus.cc:374] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.278199 18766 raft_consensus.cc:729] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d95e32416cc4bfb893efca4cdc7e218, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.278249 18766 consensus_queue.cc:260] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.278239 18767 raft_consensus.cc:348] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.278301 18767 raft_consensus.cc:374] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:53.278327 18767 raft_consensus.cc:729] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186739a517ba4a9ea54760d6b85343ef, State: Initialized, Role: FOLLOWER
I20260927 21:01:53.278337 18766 ts_tablet_manager.cc:1428] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:53.278373 18767 consensus_queue.cc:260] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.278460 18767 ts_tablet_manager.cc:1428] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:53.278462 18785 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.278548 18785 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.278750 18773 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.278870 18773 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.444651 18772 raft_consensus.cc:482] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:53.444860 18772 raft_consensus.cc:504] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.445248 18772 leader_election.cc:288] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529), 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681)
I20260927 21:01:53.445576 18785 raft_consensus.cc:482] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:53.445655 18785 raft_consensus.cc:504] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.445869 18785 leader_election.cc:288] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595), 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681)
I20260927 21:01:53.446045 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186739a517ba4a9ea54760d6b85343ef" is_pre_election: true
I20260927 21:01:53.446077 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75" is_pre_election: true
I20260927 21:01:53.446141 18697 raft_consensus.cc:2454] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 0.
I20260927 21:01:53.446193 18557 raft_consensus.cc:2454] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 0.
I20260927 21:01:53.446295 18351 leader_election.cc:302] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186739a517ba4a9ea54760d6b85343ef, 6d95e32416cc4bfb893efca4cdc7e218; no voters:
I20260927 21:01:53.446383 18785 raft_consensus.cc:2781] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:53.446431 18785 raft_consensus.cc:482] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:53.446450 18785 raft_consensus.cc:3037] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.447362 18785 raft_consensus.cc:504] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.447511 18785 leader_election.cc:288] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 election: Requested vote from peers ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595), 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681)
I20260927 21:01:53.447677 18557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad065bc1d75941cf890dee87c128da75"
I20260927 21:01:53.447680 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "6d95e32416cc4bfb893efca4cdc7e218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186739a517ba4a9ea54760d6b85343ef"
I20260927 21:01:53.447744 18697 raft_consensus.cc:3037] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.447744 18557 raft_consensus.cc:3037] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:53.448439 18557 raft_consensus.cc:2454] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 1.
I20260927 21:01:53.448489 18697 raft_consensus.cc:2454] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d95e32416cc4bfb893efca4cdc7e218 in term 1.
I20260927 21:01:53.448601 18350 leader_election.cc:302] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d95e32416cc4bfb893efca4cdc7e218, ad065bc1d75941cf890dee87c128da75; no voters:
I20260927 21:01:53.448705 18785 raft_consensus.cc:2781] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:53.448742 18785 raft_consensus.cc:686] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 LEADER]: Becoming Leader. State: Replica: 6d95e32416cc4bfb893efca4cdc7e218, State: Running, Role: LEADER
I20260927 21:01:53.448796 18785 consensus_queue.cc:237] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } }
I20260927 21:01:53.449090 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.445522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:38609 (local address 127.16.190.195:43681)
0927 21:01:53.445573 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.445575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.445779 (+ 204us) server_negotiation.cc:407] Connection header received
0927 21:01:53.445881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.445883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.445893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.445940 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.446140 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.446747 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448128 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.448651 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448707 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.448717 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.448977 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.449020 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:53.449098 18774 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: TxnStatusTablet state changed. Reason: New leader 6d95e32416cc4bfb893efca4cdc7e218. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.449177 18774 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:53.449276 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.445446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43681 (local address 127.16.190.194:38609)
0927 21:01:53.445693 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.445705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.445809 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.445983 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.445984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.445986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.446060 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.446061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.446827 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.446828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.448026 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.448028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.448794 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.448888 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.448893 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.449167 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.449196 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260927 21:01:53.449383 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.445522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43041 (local address 127.16.190.193:42529)
0927 21:01:53.445614 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.445617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.445750 (+ 133us) server_negotiation.cc:407] Connection header received
0927 21:01:53.445861 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.445863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.445874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.445927 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:53.446240 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.446977 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448227 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.448774 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.448856 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.449062 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.449101 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:01:53.449278 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.445446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42529 (local address 127.16.190.194:43041)
0927 21:01:53.445646 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.445694 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.445762 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.446069 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.446070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.446072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:53.446146 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.446148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.447079 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.447080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.448128 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.448130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.448908 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.448908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.448987 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.448989 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.449117 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.449173 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":16,"thread_start_us":92,"threads_started":1}
I20260927 21:01:53.449486 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "ad065bc1d75941cf890dee87c128da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186739a517ba4a9ea54760d6b85343ef" is_pre_election: true
I20260927 21:01:53.449571 18697 raft_consensus.cc:2379] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ad065bc1d75941cf890dee87c128da75 in current term 1: Already voted for candidate 6d95e32416cc4bfb893efca4cdc7e218 in this term.
I20260927 21:01:53.449576 18811 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:01:53.449633 18417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4138724fdcff406dac5d1391cac0bf69" candidate_uuid: "ad065bc1d75941cf890dee87c128da75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d95e32416cc4bfb893efca4cdc7e218" is_pre_election: true
I20260927 21:01:53.449697 18811 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:01:53.449605 18258 catalog_manager.cc:5495] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } health_report { overall_health: UNKNOWN } } }
I20260927 21:01:53.449882 18489 leader_election.cc:302] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad065bc1d75941cf890dee87c128da75; no voters: 186739a517ba4a9ea54760d6b85343ef, 6d95e32416cc4bfb893efca4cdc7e218
I20260927 21:01:53.450135 18772 raft_consensus.cc:2726] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 21:01:53.465106 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.462399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42529 (local address 127.0.0.1:48324)
0927 21:01:53.462464 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.462476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.462533 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.462665 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.462666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.462667 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:53.462722 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.462724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.463317 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.463318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.464255 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.464257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.464600 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.464601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.464664 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.464685 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.464925 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.464937 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.464999 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.465046 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 21:01:53.465103 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.462491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.16.190.193:42529)
0927 21:01:53.462539 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.462541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.462550 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:53.462581 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.462582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.462590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.462636 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:53.462789 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.463247 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.464344 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.464527 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.464579 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.464796 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.464853 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.464904 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.464997 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.465023 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:01:53.466223 18697 raft_consensus.cc:1261] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Refusing update from remote peer 6d95e32416cc4bfb893efca4cdc7e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.466225 18557 raft_consensus.cc:1261] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Refusing update from remote peer 6d95e32416cc4bfb893efca4cdc7e218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.466478 18774 consensus_queue.cc:1035] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.466610 18785 consensus_queue.cc:1035] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.467307 18773 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: TxnStatusTablet state changed. Reason: New leader 6d95e32416cc4bfb893efca4cdc7e218. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.467408 18773 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.467481 18772 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: TxnStatusTablet state changed. Reason: New leader 6d95e32416cc4bfb893efca4cdc7e218. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.467603 18772 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.468175 18780 mvcc.cc:205] Tried to move back new op lower bound from 7334063773555945472 to 7334063773486694400. Current Snapshot: MvccSnapshot[applied={T|T < 7334063773555945472}]
I20260927 21:01:53.468250 18785 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.468321 18785 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:53.468415 18773 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.468483 18773 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 186739a517ba4a9ea54760d6b85343ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.472468 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.469963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.16.190.195:43681)
0927 21:01:53.469999 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.470001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.470009 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:01:53.470082 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.470082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.470100 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.470164 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:53.470284 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.470733 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.471770 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.471940 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.471990 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.472161 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.472266 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.472316 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.472386 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.472404 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:01:53.477406 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.475029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:45081 (local address 127.16.190.193:42529)
0927 21:01:53.475090 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.475092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.475117 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:01:53.475188 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.475188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.475199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.475241 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:53.475391 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.475852 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.476595 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.476797 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.476910 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.477093 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.477150 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.477205 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.477308 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.477333 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:01:53.477602 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.474899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42529 (local address 127.16.190.195:45081)
0927 21:01:53.475042 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.475070 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.475134 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.475256 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.475256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.475258 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:53.475318 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.475319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.475934 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.475935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.476508 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.476510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.476912 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.476913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.476973 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.476997 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.477229 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.477231 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.477298 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.477342 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260927 21:01:53.479889 18417 raft_consensus.cc:1261] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Refusing update from remote peer 186739a517ba4a9ea54760d6b85343ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.479905 18556 raft_consensus.cc:1261] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Refusing update from remote peer 186739a517ba4a9ea54760d6b85343ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:53.480140 18797 consensus_queue.cc:1035] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.480377 18797 consensus_queue.cc:1035] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:53.481714 18785 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.481799 18785 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P 6d95e32416cc4bfb893efca4cdc7e218: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:53.482255 18772 tablet_replica.cc:439] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d95e32416cc4bfb893efca4cdc7e218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d95e32416cc4bfb893efca4cdc7e218" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42529 } } peers { permanent_uuid: "ad065bc1d75941cf890dee87c128da75" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44595 } } peers { permanent_uuid: "186739a517ba4a9ea54760d6b85343ef" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43681 } } }
I20260927 21:01:53.482338 18772 tablet_replica.cc:442] T 4138724fdcff406dac5d1391cac0bf69 P ad065bc1d75941cf890dee87c128da75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:53.490024 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.487433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:57977 (local address 127.16.190.195:43681)
0927 21:01:53.487471 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.487473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.487482 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:53.487545 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.487546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.487553 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.487596 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:53.487710 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.488163 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.489425 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.489612 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.489665 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.489842 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.489866 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.489903 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.489956 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.489972 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:01:53.490082 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.487250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43681 (local address 127.16.190.193:57977)
0927 21:01:53.487432 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:53.487444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:53.487493 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:53.487602 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:53.487602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:53.487604 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:53.487652 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.487654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.488238 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.488239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.489323 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:53.489326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.489683 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.489685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:53.489744 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.489761 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:53.489904 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:53.489906 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:01:53.489955 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:01:53.489999 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260927 21:01:53.499923 18259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:53.500042 18259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:53.500643 18259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9206e9b6fc06447b81de843ab4c74155: Sending DeleteTablet for 3 replicas of tablet 53795f7e77ba4dd2998e500e84705fc8
I20260927 21:01:53.500994 18677 tablet_service.cc:1514] Processing DeleteTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35424
I20260927 21:01:53.500974 18537 tablet_service.cc:1514] Processing DeleteTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58148
I20260927 21:01:53.501032 18397 tablet_service.cc:1514] Processing DeleteTablet for tablet 53795f7e77ba4dd2998e500e84705fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:48304
I20260927 21:01:53.501080 18791 tablet_replica.cc:330] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: stopping tablet replica
I20260927 21:01:53.501091 18792 tablet_replica.cc:330] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: stopping tablet replica
I20260927 21:01:53.501135 18791 raft_consensus.cc:2229] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:53.501144 18792 raft_consensus.cc:2229] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:53.501180 18791 raft_consensus.cc:2258] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.501255 18792 raft_consensus.cc:2258] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.501260 18790 tablet_replica.cc:330] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: stopping tablet replica
I20260927 21:01:53.501315 18790 raft_consensus.cc:2229] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:53.501348 18790 raft_consensus.cc:2258] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:53.501663 18790 ts_tablet_manager.cc:1905] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.501641 18791 ts_tablet_manager.cc:1905] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.503223 18792 ts_tablet_manager.cc:1905] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:53.503470 18790 ts_tablet_manager.cc:1918] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:53.503471 18791 ts_tablet_manager.cc:1918] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:53.503527 18791 log.cc:1196] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/53795f7e77ba4dd2998e500e84705fc8
I20260927 21:01:53.503527 18790 log.cc:1196] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/53795f7e77ba4dd2998e500e84705fc8
I20260927 21:01:53.503750 18790 ts_tablet_manager.cc:1939] T 53795f7e77ba4dd2998e500e84705fc8 P 6d95e32416cc4bfb893efca4cdc7e218: Deleting consensus metadata
I20260927 21:01:53.503750 18791 ts_tablet_manager.cc:1939] T 53795f7e77ba4dd2998e500e84705fc8 P ad065bc1d75941cf890dee87c128da75: Deleting consensus metadata
I20260927 21:01:53.504040 18246 catalog_manager.cc:4841] TS ad065bc1d75941cf890dee87c128da75 (127.16.190.194:44595): tablet 53795f7e77ba4dd2998e500e84705fc8 (table test-table [id=6f78327e8feb492ebe54ee0173ed059c]) successfully deleted
I20260927 21:01:53.504040 18245 catalog_manager.cc:4841] TS 6d95e32416cc4bfb893efca4cdc7e218 (127.16.190.193:42529): tablet 53795f7e77ba4dd2998e500e84705fc8 (table test-table [id=6f78327e8feb492ebe54ee0173ed059c]) successfully deleted
I20260927 21:01:53.504714 18792 ts_tablet_manager.cc:1918] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:53.504771 18792 log.cc:1196] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/53795f7e77ba4dd2998e500e84705fc8
I20260927 21:01:53.505002 18792 ts_tablet_manager.cc:1939] T 53795f7e77ba4dd2998e500e84705fc8 P 186739a517ba4a9ea54760d6b85343ef: Deleting consensus metadata
I20260927 21:01:53.505509 18247 catalog_manager.cc:4841] TS 186739a517ba4a9ea54760d6b85343ef (127.16.190.195:43681): tablet 53795f7e77ba4dd2998e500e84705fc8 (table test-table [id=6f78327e8feb492ebe54ee0173ed059c]) successfully deleted
Sep 27 21:01:53 dist-test-slave-6c7s krb5kdc[18215](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:01:53.512465 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.502045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.16.190.193:42529)
0927 21:01:53.502089 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.502092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.502255 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:01:53.502303 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.502304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.502318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.502377 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:53.503089 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.503827 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.504989 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.505336 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.505403 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.509235 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:53.509245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:53.509246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:53.509254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:53.511192 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.511556 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.511559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.511559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.511616 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.511810 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.511811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.511811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.511958 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:53.512061 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.512288 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.512370 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:01:53.513015 18457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48326
I20260927 21:01:53.521760 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:53.513946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.16.190.254:36155)
0927 21:01:53.513992 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:53.513994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:53.514221 (+ 227us) server_negotiation.cc:407] Connection header received
0927 21:01:53.514274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:53.514275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:53.514301 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:53.514363 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:53.514989 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.515793 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.516904 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:53.517228 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:53.517306 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:53.518780 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:53.518782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:53.518783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:53.518792 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:53.520509 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.520943 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.520945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.520945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.521010 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:53.521224 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:53.521224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:53.521225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:53.521365 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:53.521467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:53.521580 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:01:53.521659 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 21:01:53.522070 18259 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35636
I20260927 21:01:53.522838 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18332
I20260927 21:01:53.528962 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18472
I20260927 21:01:53.534181 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18612
I20260927 21:01:53.539525 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18228
2026-09-27T21:01:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2588 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:01:53 dist-test-slave-6c7s krb5kdc[18832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:01:53 dist-test-slave-6c7s krb5kdc[18832](info): set up 2 sockets
Sep 27 21:01:53 dist-test-slave-6c7s krb5kdc[18832](info): commencing operation
krb5kdc: starting...
W20260927 21:01:54.577255 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:01:54 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:01:54.700106 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37653
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:54.781474 18845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:54.781697 18845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:54.781720 18845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:54.782903 18845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:01:54.782948 18845 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:01:54.782963 18845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:54.782975 18845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:54.782987 18845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:01:54.782999 18845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:01:54.784499 18845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37653
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:54.784762 18845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:54.785013 18845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:54.789996 18851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:54.790046 18850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:54.789996 18853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:54.790294 18845 server_base.cc:1034] running on GCE node
I20260927 21:01:54.790544 18845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:54.790848 18845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:54.792013 18845 hybrid_clock.cc:648] HybridClock initialized: now 1790542914792001 us; error 29 us; skew 500 ppm
Sep 27 21:01:54 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542914, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:54.794628 18845 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:01:54.795037 18845 webserver.cc:460] Webserver started at http://127.16.190.254:41231/ using document root <none> and password file <none>
I20260927 21:01:54.795231 18845 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:54.795272 18845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:54.795357 18845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:54.796371 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "9c4592054a004e979744b9cb50da8662"
format_stamp: "Formatted at 2026-09-27 21:01:54 on dist-test-slave-6c7s"
server_key: "86f805704d201c7cb2ea2fc6c7ad587c"
server_key_iv: "ae3faa546e47c845c3baf70fc0646566"
server_key_version: "encryptionkey@0"
I20260927 21:01:54.796687 18845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "9c4592054a004e979744b9cb50da8662"
format_stamp: "Formatted at 2026-09-27 21:01:54 on dist-test-slave-6c7s"
server_key: "86f805704d201c7cb2ea2fc6c7ad587c"
server_key_iv: "ae3faa546e47c845c3baf70fc0646566"
server_key_version: "encryptionkey@0"
I20260927 21:01:54.798144 18845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:01:54.798877 18860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:54.799057 18845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:01:54.799191 18845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "9c4592054a004e979744b9cb50da8662"
format_stamp: "Formatted at 2026-09-27 21:01:54 on dist-test-slave-6c7s"
server_key: "86f805704d201c7cb2ea2fc6c7ad587c"
server_key_iv: "ae3faa546e47c845c3baf70fc0646566"
server_key_version: "encryptionkey@0"
I20260927 21:01:54.799270 18845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:54.822575 18845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:54.824640 18845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:54.824795 18845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:54.829306 18845 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:37653
I20260927 21:01:54.829380 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:37653 every 8 connection(s)
I20260927 21:01:54.829738 18845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:01:54.830446 18923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:54.832753 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Bootstrap starting.
I20260927 21:01:54.833359 18923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:54.833652 18923 log.cc:824] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:54.834328 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: No bootstrap required, opened a new log
I20260927 21:01:54.835197 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18845
I20260927 21:01:54.835290 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:01:54.835438 17147 external_mini_cluster.cc:1428] Setting key acd22f5a670a365698c005eced877256
I20260927 21:01:54.836438 18923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } }
I20260927 21:01:54.836603 18923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:54.836715 18923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c4592054a004e979744b9cb50da8662, State: Initialized, Role: FOLLOWER
I20260927 21:01:54.836889 18923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } }
I20260927 21:01:54.836985 18923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:01:54.837026 18923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:01:54.837081 18923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:54.837747 18923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } }
I20260927 21:01:54.837867 18923 leader_election.cc:302] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c4592054a004e979744b9cb50da8662; no voters:
I20260927 21:01:54.838037 18923 leader_election.cc:288] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:01:54.838090 18928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:54.838279 18928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [term 1 LEADER]: Becoming Leader. State: Replica: 9c4592054a004e979744b9cb50da8662, State: Running, Role: LEADER
I20260927 21:01:54.838297 18923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:01:54.838415 18928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } }
I20260927 21:01:54.838765 18930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c4592054a004e979744b9cb50da8662. Latest consensus state: current_term: 1 leader_uuid: "9c4592054a004e979744b9cb50da8662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } } }
I20260927 21:01:54.838852 18930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [sys.catalog]: This master's current role is: LEADER
I20260927 21:01:54.838999 18929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c4592054a004e979744b9cb50da8662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c4592054a004e979744b9cb50da8662" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37653 } } }
I20260927 21:01:54.839056 18929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662 [sys.catalog]: This master's current role is: LEADER
W20260927 21:01:54.840209 18943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:01:54.840308 18943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:01:54.840378 18935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:01:54.840552 18935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:01:54.842198 18935 catalog_manager.cc:1347] Generated new cluster ID: b3a67178e2a9410dad158faf2b36a8f0
I20260927 21:01:54.842252 18935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:01:54 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:54.846199 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:54.835923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.16.190.254:37653)
0927 21:01:54.836325 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:54.836333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:54.836354 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:01:54.836452 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:54.836457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:54.836509 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:54.836655 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:54.837425 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:54.838700 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:54.839686 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:54.840345 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:54.840452 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:54.843109 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:54.843114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:54.843114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:54.843125 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:54.844623 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:54.845062 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:54.845064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:54.845065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:54.845158 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:54.845369 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:54.845369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:54.845370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:54.845605 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:54.845772 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:54.845895 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:01:54.846035 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260927 21:01:54.864490 18935 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:01:54.865160 18935 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:01:54.869622 18935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Generated new TSK 0
I20260927 21:01:54.869841 18935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:01:54.903815 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37653
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:55.006390 18949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:55.006631 18949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:55.006675 18949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:55.007853 18949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:55.007920 18949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:55.007957 18949 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:01:55.007985 18949 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:55.008019 18949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:01:55.009692 18949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:55.009960 18949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:55.010208 18949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:55.015534 18957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.015558 18955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.015540 18954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:55.015816 18949 server_base.cc:1034] running on GCE node
I20260927 21:01:55.016003 18949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:55.016229 18949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:55.017383 18949 hybrid_clock.cc:648] HybridClock initialized: now 1790542915017348 us; error 51 us; skew 500 ppm
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:55.020233 18949 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:01:55.020787 18949 webserver.cc:460] Webserver started at http://127.16.190.193:46391/ using document root <none> and password file <none>
I20260927 21:01:55.021060 18949 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:55.021130 18949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:55.021283 18949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:55.022353 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "910a7788dcce2eb517b41f0d0f84134d"
server_key_iv: "868da8bc24d15bb7c2183c5248392630"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.022751 18949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "910a7788dcce2eb517b41f0d0f84134d"
server_key_iv: "868da8bc24d15bb7c2183c5248392630"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.024166 18949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:01:55.024914 18964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.025089 18949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:55.025189 18949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "910a7788dcce2eb517b41f0d0f84134d"
server_key_iv: "868da8bc24d15bb7c2183c5248392630"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.025275 18949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:55.042861 18949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:55.045166 18949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:55.045357 18949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:55.046030 18949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:55.046087 18949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.046125 18949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:55.046149 18949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:55.053463 18949 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42249
I20260927 21:01:55.053560 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42249 every 8 connection(s)
I20260927 21:01:55.053855 18949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:01:55.056879 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.046445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:44701 (local address 127.16.190.254:37653)
0927 21:01:55.046515 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.046518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.046843 (+ 325us) server_negotiation.cc:407] Connection header received
0927 21:01:55.047625 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.047627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.047643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.047716 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.048675 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.049289 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.051022 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.051392 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.051461 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.053952 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.053954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.053955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.053963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.055454 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.055875 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.055876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.055876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.055930 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.056220 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.056221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.056221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.056330 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.056385 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.056667 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.056752 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:01:55.057030 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.046439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.193:44701)
0927 21:01:55.046657 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.046713 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.047462 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.047807 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.047811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.048221 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.048556 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.048565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.049367 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.049368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.050901 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.050905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.051481 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.051483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.051639 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.052185 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.052195 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.053838 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.055568 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.055580 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.055582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.055808 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.056018 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.056019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.056019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.056130 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.056401 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.056406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.056527 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.056616 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.056745 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260927 21:01:55.057583 19080 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37653
I20260927 21:01:55.057717 19080 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:55.057926 19080 heartbeater.cc:511] Master 127.16.190.254:37653 requested a full tablet report, sending...
I20260927 21:01:55.058449 18877 ts_manager.cc:194] Registered new tserver with Master: 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249)
I20260927 21:01:55.059408 18877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:44701
I20260927 21:01:55.059779 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 18949
I20260927 21:01:55.059870 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:01:55.060066 17147 external_mini_cluster.cc:1428] Setting key bb205da2f6e4049f3d9e352725ae3967
I20260927 21:01:55.064937 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.058637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.193:37809)
0927 21:01:55.058797 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.058809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.058873 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.058989 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.058990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.059183 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.059248 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.059250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.060123 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.060124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.060717 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.060720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.061119 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.061120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.061208 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.061626 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.061632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.062090 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.063853 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.063854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.063855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.064015 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.064222 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.064222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.064223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.064272 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.064573 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.064574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.064616 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.064693 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.064777 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260927 21:01:55.064976 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.058707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:37809 (local address 127.16.190.254:37653)
0927 21:01:55.058749 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.058751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.058878 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:01:55.058928 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.058929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.058936 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.058976 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.059335 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.060015 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.060798 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.061046 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.061099 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.062188 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.062190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.062190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.062198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.063774 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.064087 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.064088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.064088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.064148 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.064342 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.064343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.064343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.064483 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.064559 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.064742 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.064869 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:01:55.088939 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37653
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:55.170310 19089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:55.170527 19089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:55.170552 19089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:55.171690 19089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:55.171732 19089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:55.171752 19089 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:01:55.171766 19089 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:55.171783 19089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:01:55.173252 19089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:55.173488 19089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:55.173748 19089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:55.178742 19095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.178757 19097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.178759 19094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:55.178951 19089 server_base.cc:1034] running on GCE node
I20260927 21:01:55.179168 19089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:55.179363 19089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:55.180496 19089 hybrid_clock.cc:648] HybridClock initialized: now 1790542915180484 us; error 29 us; skew 500 ppm
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:55.182941 19089 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:01:55.183431 19089 webserver.cc:460] Webserver started at http://127.16.190.194:40153/ using document root <none> and password file <none>
I20260927 21:01:55.183650 19089 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:55.183700 19089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:55.183861 19089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:55.185026 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "0fa43adda6064881ba08c1b2902130e2"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "ffe3403847f0f5c3b6842868cceacba4"
server_key_iv: "36f09cee00f3875faa9681221d74bc98"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.185392 19089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "0fa43adda6064881ba08c1b2902130e2"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "ffe3403847f0f5c3b6842868cceacba4"
server_key_iv: "36f09cee00f3875faa9681221d74bc98"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.186806 19089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:55.187647 19104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.187825 19089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:55.187932 19089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "0fa43adda6064881ba08c1b2902130e2"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "ffe3403847f0f5c3b6842868cceacba4"
server_key_iv: "36f09cee00f3875faa9681221d74bc98"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.188007 19089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:55.203584 19089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:55.205643 19089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:55.205821 19089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:55.206420 19089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:55.206470 19089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.206501 19089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:55.206521 19089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:55.213680 19089 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:37765
I20260927 21:01:55.213759 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:37765 every 8 connection(s)
I20260927 21:01:55.214068 19089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:01:55.216097 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.206863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:33383 (local address 127.16.190.254:37653)
0927 21:01:55.206933 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.206935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.207184 (+ 249us) server_negotiation.cc:407] Connection header received
0927 21:01:55.207663 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.207665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.207680 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.207763 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.208479 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.209133 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.210151 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.210346 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.210421 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.213151 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.213153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.213153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.213162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.214796 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.215175 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.215176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.215177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.215237 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.215529 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.215531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.215531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.215645 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.215704 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.215947 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.216029 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:01:55.216235 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.206862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.194:33383)
0927 21:01:55.207035 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.207081 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.207567 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.207864 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.207868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.208184 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.208374 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.208380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.209227 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.209228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.210031 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.210033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.210440 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.210441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.210561 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.211081 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.211091 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.213053 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.214866 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.214871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.214872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.215087 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.215328 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.215329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.215330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.215451 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.215708 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.215712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.215824 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.215909 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.216037 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
I20260927 21:01:55.216769 19220 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37653
I20260927 21:01:55.216871 19220 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:55.217065 19220 heartbeater.cc:511] Master 127.16.190.254:37653 requested a full tablet report, sending...
I20260927 21:01:55.217446 18876 ts_manager.cc:194] Registered new tserver with Master: 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765)
I20260927 21:01:55.217983 18876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:33383
I20260927 21:01:55.223449 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.217972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51613 (local address 127.16.190.254:37653)
0927 21:01:55.218012 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.218014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.218099 (+ 85us) server_negotiation.cc:407] Connection header received
0927 21:01:55.218176 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.218177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.218187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.218236 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.218601 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.219070 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.219784 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.219982 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.220056 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.221039 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.221040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.221041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.221047 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.222189 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.222554 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.222555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.222555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.222612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.222870 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.222871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.222871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.223017 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.223097 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.223265 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.223359 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:55.223489 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.217877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.194:51613)
0927 21:01:55.218025 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.218038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.218125 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.218238 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.218239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.218426 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.218516 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.218518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.219167 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.219168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.219689 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.219691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.220054 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.220055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.220133 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.220545 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.220549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.220928 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.222294 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.222295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.222295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.222471 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.222723 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.222724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.222724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.222789 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.223088 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.223089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.223135 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.223239 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.223384 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
I20260927 21:01:55.223851 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19089
I20260927 21:01:55.223978 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:01:55.224175 17147 external_mini_cluster.cc:1428] Setting key d5c96a126ddadfe99cae0242e6c0e18e
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:01:55.252154 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37653
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:01:55.332707 19229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:01:55.332974 19229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:01:55.333019 19229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:01:55.334187 19229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:01:55.334251 19229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:01:55.334288 19229 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 21:01:55.334334 19229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:01:55.334368 19229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:01:55.335794 19229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:01:55.336061 19229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:01:55.336302 19229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:01:55.341543 19235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.341524 19237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:01:55.341730 19234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:01:55.341876 19229 server_base.cc:1034] running on GCE node
I20260927 21:01:55.342087 19229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:01:55.342326 19229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:01:55.343503 19229 hybrid_clock.cc:648] HybridClock initialized: now 1790542915343487 us; error 24 us; skew 500 ppm
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:01:55.346176 19229 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:01:55.346652 19229 webserver.cc:460] Webserver started at http://127.16.190.195:46133/ using document root <none> and password file <none>
I20260927 21:01:55.346870 19229 fs_manager.cc:359] Metadata directory not provided
I20260927 21:01:55.346913 19229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:01:55.347093 19229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:01:55.348162 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "c106b6fa21cf42c2846c91019d191670"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "19749f1b6f8e8c1e112ee15758d52acc"
server_key_iv: "9fa45a7f97bde1e3ff3f0f34b5832cf2"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.348526 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "c106b6fa21cf42c2846c91019d191670"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "19749f1b6f8e8c1e112ee15758d52acc"
server_key_iv: "9fa45a7f97bde1e3ff3f0f34b5832cf2"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.349946 19229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:01:55.350749 19244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.350929 19229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:55.351042 19229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "c106b6fa21cf42c2846c91019d191670"
format_stamp: "Formatted at 2026-09-27 21:01:55 on dist-test-slave-6c7s"
server_key: "19749f1b6f8e8c1e112ee15758d52acc"
server_key_iv: "9fa45a7f97bde1e3ff3f0f34b5832cf2"
server_key_version: "encryptionkey@0"
I20260927 21:01:55.351116 19229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:01:55.367594 19229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:01:55.370185 19229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:01:55.370352 19229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:01:55.370872 19229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:01:55.370934 19229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.370980 19229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:01:55.371017 19229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:55.378552 19229 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:42867
I20260927 21:01:55.378643 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:42867 every 8 connection(s)
I20260927 21:01:55.378960 19229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:01:55.381870 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.371398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.195:50791)
0927 21:01:55.371569 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.371610 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.372109 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.372402 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.372405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.372829 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.373049 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.373055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.374094 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.374095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.375231 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.375235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.375783 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.375785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.375962 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.376576 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.376587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.378553 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.380350 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.380355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.380357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.380619 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.380801 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.380801 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.380802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.380965 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.381293 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.381298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.381433 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.381532 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.381708 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260927 21:01:55.381958 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.371407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:50791 (local address 127.16.190.254:37653)
0927 21:01:55.371461 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.371465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.371709 (+ 244us) server_negotiation.cc:407] Connection header received
0927 21:01:55.372203 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.372204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.372217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.372295 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.373147 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.373995 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.375347 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.375678 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.375754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.378672 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.378674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.378674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.378682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.380274 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.380694 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.380695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.380695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.380745 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.381048 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.381050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.381050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.381183 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.381265 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.381577 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.381838 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:01:55.382344 19360 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37653
I20260927 21:01:55.382443 19360 heartbeater.cc:464] Registering TS with master...
I20260927 21:01:55.382639 19360 heartbeater.cc:511] Master 127.16.190.254:37653 requested a full tablet report, sending...
I20260927 21:01:55.383024 18877 ts_manager.cc:194] Registered new tserver with Master: c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:55.383487 18877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:50791
I20260927 21:01:55.387135 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19229
I20260927 21:01:55.387241 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:01:55.387416 17147 external_mini_cluster.cc:1428] Setting key 335eb53145a4a6343b04cb7d72ff00e6
I20260927 21:01:55.388693 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:01:55.390439 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.383608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:32775 (local address 127.16.190.254:37653)
0927 21:01:55.383648 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.383650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.383828 (+ 178us) server_negotiation.cc:407] Connection header received
0927 21:01:55.383895 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.383895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.383905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.383951 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.384340 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.384938 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.385696 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.385882 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.385958 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.387196 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.387198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.387198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.387205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.389145 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.389539 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.389541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.389541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.389607 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.389847 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.389848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.389848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.389970 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.390045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.390231 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.390334 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:01:55.390456 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.383560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.16.190.195:32775)
0927 21:01:55.383767 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.383778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.383837 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.383965 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.383966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.384168 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.384239 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.384241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.385052 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.385053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.385567 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.385569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.385975 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.385976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.386044 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.386703 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.386708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.387055 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.389246 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.389247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.389248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.389461 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.389688 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.389688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.389689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.389770 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.390036 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.390037 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.390088 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.390164 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.390304 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:55.413187 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.405561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.16.190.254:37653)
0927 21:01:55.405617 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.405619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.405651 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:01:55.405719 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.405720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.405730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.405779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.406335 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.406840 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.407740 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.407930 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.407995 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.410481 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.410482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.410483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.410490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.411848 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.412216 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.412217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.412217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.412271 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.412469 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.412469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.412469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.412573 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.412646 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.412948 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.413065 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:55.415189 18877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:55.416924 18877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:55.424099 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.420043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.16.190.194:37765)
0927 21:01:55.420260 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.420265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.420276 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:01:55.420311 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.420313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.420325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.420425 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:55.420633 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421827 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.423001 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.423698 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.423782 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.423838 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.423963 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.424006 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
I20260927 21:01:55.424090 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.419797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37765 (local address 127.0.0.1:35042)
0927 21:01:55.420203 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.420213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.420254 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.420450 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.420455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.420457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:55.420533 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.420535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421918 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.421919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.422900 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.422902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.423781 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.423782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.423845 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.423857 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.423945 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.423999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":95,"threads_started":1}
I20260927 21:01:55.424374 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.419775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42249 (local address 127.0.0.1:56082)
0927 21:01:55.420148 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.420159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.420219 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.420722 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.420723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.420725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:55.420774 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.420782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421784 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.421785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.423162 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.423166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.424078 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.424079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.424139 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.424143 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.424172 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.424299 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":83,"thread_start_us":110,"threads_started":1}
I20260927 21:01:55.424798 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.419972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.16.190.193:42249)
0927 21:01:55.420554 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.420558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.420567 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:55.420604 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.420606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.420621 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.420903 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:55.420961 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421698 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.423252 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.423985 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.424548 (+ 563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.424620 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.424646 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.424686 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":75,"threads_started":1}
I20260927 21:01:55.425282 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.419775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:42867 (local address 127.0.0.1:52614)
0927 21:01:55.419944 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.420010 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.420081 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.420569 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.420570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.420572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:55.420615 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.420616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421896 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.421897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.423164 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.423166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.424858 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.424859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.425092 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.425097 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.425174 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.425217 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":72,"thread_start_us":49,"threads_started":1}
I20260927 21:01:55.425169 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 8286f3eb9a43420fbe70069ef10630ee (DEFAULT_TABLE table=test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.425336 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.420038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.16.190.195:42867)
0927 21:01:55.420332 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.420337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.420346 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:01:55.420391 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.420413 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.420429 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.420504 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:55.420693 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.421815 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.423889 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.424681 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.424745 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.424792 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.425202 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.425244 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":82,"threads_started":1}
I20260927 21:01:55.425443 19014 tablet_service.cc:1467] Processing CreateTablet for tablet 8286f3eb9a43420fbe70069ef10630ee (DEFAULT_TABLE table=test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.425576 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8286f3eb9a43420fbe70069ef10630ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.425719 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8286f3eb9a43420fbe70069ef10630ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.428212 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 8286f3eb9a43420fbe70069ef10630ee (DEFAULT_TABLE table=test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.428516 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8286f3eb9a43420fbe70069ef10630ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.428598 19383 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Bootstrap starting.
I20260927 21:01:55.429250 19383 tablet_bootstrap.cc:654] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.429556 19383 log.cc:824] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:55.430241 19383 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: No bootstrap required, opened a new log
I20260927 21:01:55.430295 19383 ts_tablet_manager.cc:1397] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:55.431187 19385 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Bootstrap starting.
I20260927 21:01:55.432159 19385 tablet_bootstrap.cc:654] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.432132 19386 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Bootstrap starting.
I20260927 21:01:55.432190 19383 raft_consensus.cc:348] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.432379 19383 raft_consensus.cc:374] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.432416 19383 raft_consensus.cc:729] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9b9cae411b4ec5954b8797e4b9869b, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.432487 19385 log.cc:824] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:55.432567 19383 consensus_queue.cc:260] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.432854 19383 ts_tablet_manager.cc:1428] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:01:55.432966 19386 tablet_bootstrap.cc:654] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.433005 19080 heartbeater.cc:503] Master 127.16.190.254:37653 was elected leader, sending a full tablet report...
I20260927 21:01:55.433315 19386 log.cc:824] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:01:55.433588 19385 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: No bootstrap required, opened a new log
I20260927 21:01:55.433674 19385 ts_tablet_manager.cc:1397] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:01:55.434042 19386 tablet_bootstrap.cc:492] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: No bootstrap required, opened a new log
I20260927 21:01:55.434154 19386 ts_tablet_manager.cc:1397] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:01:55.435166 19385 raft_consensus.cc:348] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.435324 19385 raft_consensus.cc:374] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.435371 19385 raft_consensus.cc:729] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.435523 19385 consensus_queue.cc:260] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.435820 19385 ts_tablet_manager.cc:1428] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:01:55.435729 19386 raft_consensus.cc:348] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.435835 19360 heartbeater.cc:503] Master 127.16.190.254:37653 was elected leader, sending a full tablet report...
I20260927 21:01:55.435894 19386 raft_consensus.cc:374] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.435930 19386 raft_consensus.cc:729] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.436045 19386 consensus_queue.cc:260] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.436336 19386 ts_tablet_manager.cc:1428] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:01:55.436381 19220 heartbeater.cc:503] Master 127.16.190.254:37653 was elected leader, sending a full tablet report...
W20260927 21:01:55.464717 19221 tablet.cc:2369] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:01:55.504688 19391 raft_consensus.cc:482] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:55.504860 19391 raft_consensus.cc:504] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.505256 19391 leader_election.cc:288] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867), 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249)
I20260927 21:01:55.508308 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.505573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:48229 (local address 127.16.190.195:42867)
0927 21:01:55.505649 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.505652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.505665 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:01:55.505738 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.505739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.505749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.505829 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:55.506014 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.506527 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.507539 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.507924 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.508011 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.508020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.508192 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.508231 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 21:01:55.508308 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.505485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:42867 (local address 127.16.190.194:48229)
0927 21:01:55.505551 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.505565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.505673 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.505852 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.505853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.505854 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:55.505939 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.505941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.506624 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.506625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.507438 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.507440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.508018 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.508019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.508113 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.508116 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.508186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.508229 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":26}
I20260927 21:01:55.508375 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.505642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:40037 (local address 127.16.190.193:42249)
0927 21:01:55.505686 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.505687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.505819 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:01:55.505918 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.505919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.505930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.506002 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:55.506175 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.506656 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.507608 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.508003 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.508089 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.508099 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.508260 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.508298 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:01:55.508440 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.505502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42249 (local address 127.16.190.194:40037)
0927 21:01:55.505732 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.505749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.505827 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.506005 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.506006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.506008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:55.506100 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.506102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.506748 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.506749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.507489 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.507491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.508110 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.508111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.508184 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.508187 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.508255 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.508298 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":171,"threads_started":1}
I20260927 21:01:55.508882 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8286f3eb9a43420fbe70069ef10630ee" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" is_pre_election: true
I20260927 21:01:55.508875 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8286f3eb9a43420fbe70069ef10630ee" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670" is_pre_election: true
I20260927 21:01:55.509056 19034 raft_consensus.cc:2454] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.509048 19312 raft_consensus.cc:2454] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.509326 19107 leader_election.cc:302] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9b9cae411b4ec5954b8797e4b9869b, 0fa43adda6064881ba08c1b2902130e2; no voters:
I20260927 21:01:55.509486 19391 raft_consensus.cc:2781] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:55.509543 19391 raft_consensus.cc:482] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:55.509581 19391 raft_consensus.cc:3037] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.510515 19391 raft_consensus.cc:504] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.510679 19391 leader_election.cc:288] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Requested vote from peers c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867), 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249)
I20260927 21:01:55.510854 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8286f3eb9a43420fbe70069ef10630ee" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670"
I20260927 21:01:55.510861 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8286f3eb9a43420fbe70069ef10630ee" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
I20260927 21:01:55.510943 19312 raft_consensus.cc:3037] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.510964 19034 raft_consensus.cc:3037] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.511693 19034 raft_consensus.cc:2454] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.511713 19312 raft_consensus.cc:2454] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.511929 19106 leader_election.cc:302] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:55.512014 19391 raft_consensus.cc:2781] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:55.512212 19391 raft_consensus.cc:686] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Running, Role: LEADER
I20260927 21:01:55.512315 19391 consensus_queue.cc:237] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } }
I20260927 21:01:55.513167 18876 catalog_manager.cc:5495] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260927 21:01:55.554948 19081 tablet.cc:2369] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 21:01:55 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542914, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:55.568584 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.561504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37653 (local address 127.0.0.1:48400)
0927 21:01:55.561729 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.561739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.561827 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.561992 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.561993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.562323 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:01:55.562370 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.562373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.563029 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.563030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.563547 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.563548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.563904 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.563905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.563954 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.564480 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:01:55.564485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:01:55.565938 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:01:55.567446 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.567455 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.567457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.567661 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.567867 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:01:55.567868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:01:55.567869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:01:55.567931 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:01:55.568187 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:01:55.568195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:01:55.568240 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.568321 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.568447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":130,"threads_started":1}
I20260927 21:01:55.568585 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.561579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.16.190.254:37653)
0927 21:01:55.561666 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.561668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.561806 (+ 138us) server_negotiation.cc:407] Connection header received
0927 21:01:55.561914 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.561914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.561923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.561987 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:55.562465 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.562948 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.563653 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.563833 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.563907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.566045 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:55.566046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:55.566047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:55.566054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:55.567347 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.567736 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.567737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.567738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.567793 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:55.567991 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:55.567993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:55.567993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:55.568116 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:55.568188 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.568359 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.568515 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":37}
I20260927 21:01:55.570657 18876 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:48400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:01:55.570901 18876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:55.572449 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 483f274661694e889060e9b1eebcfd61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97312be08d3c47cb8cbb719a969eda48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:55.572439 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 483f274661694e889060e9b1eebcfd61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97312be08d3c47cb8cbb719a969eda48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:55.572446 19014 tablet_service.cc:1467] Processing CreateTablet for tablet 483f274661694e889060e9b1eebcfd61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97312be08d3c47cb8cbb719a969eda48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:01:55.572629 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483f274661694e889060e9b1eebcfd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.572633 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483f274661694e889060e9b1eebcfd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.572659 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483f274661694e889060e9b1eebcfd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.573971 19385 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: Bootstrap starting.
I20260927 21:01:55.574172 19383 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: Bootstrap starting.
I20260927 21:01:55.574479 19383 tablet_bootstrap.cc:654] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.574515 19385 tablet_bootstrap.cc:654] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.574775 19386 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: Bootstrap starting.
I20260927 21:01:55.575130 19385 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: No bootstrap required, opened a new log
I20260927 21:01:55.575172 19385 ts_tablet_manager.cc:1397] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:55.575191 19386 tablet_bootstrap.cc:654] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.575342 19385 raft_consensus.cc:348] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.575431 19385 raft_consensus.cc:374] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.575457 19385 raft_consensus.cc:729] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.575507 19385 consensus_queue.cc:260] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.575620 19385 ts_tablet_manager.cc:1428] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:55.575764 19383 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: No bootstrap required, opened a new log
I20260927 21:01:55.575812 19383 ts_tablet_manager.cc:1397] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:55.575758 19390 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.575848 19390 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.575970 19383 raft_consensus.cc:348] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.576052 19383 raft_consensus.cc:374] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.576082 19383 raft_consensus.cc:729] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9b9cae411b4ec5954b8797e4b9869b, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.576148 19383 consensus_queue.cc:260] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.576278 19383 ts_tablet_manager.cc:1428] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:55.576318 19388 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.576392 19388 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.576736 19386 tablet_bootstrap.cc:492] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: No bootstrap required, opened a new log
I20260927 21:01:55.576804 19386 ts_tablet_manager.cc:1397] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:01:55.576994 19386 raft_consensus.cc:348] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.577109 19386 raft_consensus.cc:374] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.577135 19386 raft_consensus.cc:729] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.577188 19386 consensus_queue.cc:260] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.577281 19386 ts_tablet_manager.cc:1428] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.577359 19391 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.577466 19391 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 21:01:55.629657 19361 tablet.cc:2369] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:01:55.656757 19391 raft_consensus.cc:482] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:55.656921 19391 raft_consensus.cc:504] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.657140 19391 leader_election.cc:288] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:55.657405 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483f274661694e889060e9b1eebcfd61" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670" is_pre_election: true
I20260927 21:01:55.657433 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483f274661694e889060e9b1eebcfd61" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" is_pre_election: true
I20260927 21:01:55.657528 19312 raft_consensus.cc:2454] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.657528 19034 raft_consensus.cc:2454] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.657691 19106 leader_election.cc:302] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:55.657819 19391 raft_consensus.cc:2781] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:55.657855 19391 raft_consensus.cc:482] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:55.657874 19391 raft_consensus.cc:3037] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.658603 19391 raft_consensus.cc:504] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.658772 19391 leader_election.cc:288] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:55.658919 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483f274661694e889060e9b1eebcfd61" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
I20260927 21:01:55.658972 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483f274661694e889060e9b1eebcfd61" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670"
I20260927 21:01:55.659016 19034 raft_consensus.cc:3037] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.659037 19312 raft_consensus.cc:3037] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.659804 19034 raft_consensus.cc:2454] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.659804 19312 raft_consensus.cc:2454] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.660009 19106 leader_election.cc:302] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:55.660121 19391 raft_consensus.cc:2781] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:55.660195 19391 raft_consensus.cc:686] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Running, Role: LEADER
I20260927 21:01:55.660256 19391 consensus_queue.cc:237] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.660657 19393 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: TxnStatusTablet state changed. Reason: New leader 0fa43adda6064881ba08c1b2902130e2. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.660782 19393 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:55.661079 18876 catalog_manager.cc:5495] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } health_report { overall_health: UNKNOWN } } }
I20260927 21:01:55.661383 19405 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:01:55.661544 19405 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:01:55.673167 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.670635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.16.190.194:37765)
0927 21:01:55.670691 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.670694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.670705 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:01:55.670789 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.670790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.670808 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.670868 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:55.670996 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.671483 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.672260 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.672476 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.672551 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.672756 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:55.672925 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:55.673017 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.673060 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.673081 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:01:55.673180 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.670566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37765 (local address 127.0.0.1:35044)
0927 21:01:55.670641 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.670653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.670715 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.670881 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.670882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.670883 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:55.670940 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.670943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.671568 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.671569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.672166 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.672168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.672560 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.672562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.672628 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.672647 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:55.672995 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:55.673006 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.673057 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.673110 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:01:55.675132 19312 raft_consensus.cc:1261] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.675136 19034 raft_consensus.cc:1261] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.675464 19393 consensus_queue.cc:1035] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:55.675642 19391 consensus_queue.cc:1035] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:55.676826 19390 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: TxnStatusTablet state changed. Reason: New leader 0fa43adda6064881ba08c1b2902130e2. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.676944 19390 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.677078 19388 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: TxnStatusTablet state changed. Reason: New leader 0fa43adda6064881ba08c1b2902130e2. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.677168 19388 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.678099 19391 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.678217 19391 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:55.678323 19391 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.678391 19391 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0fa43adda6064881ba08c1b2902130e2: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:01:55.678682 19408 mvcc.cc:205] Tried to move back new op lower bound from 7334063782602747904 to 7334063782544789504. Current Snapshot: MvccSnapshot[applied={T|T < 7334063782602747904}]
I20260927 21:01:55.678723 19388 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.678810 19388 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P 0b9b9cae411b4ec5954b8797e4b9869b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.679035 19409 mvcc.cc:205] Tried to move back new op lower bound from 7334063782602747904 to 7334063782544789504. Current Snapshot: MvccSnapshot[applied={T|T < 7334063782602747904}]
I20260927 21:01:55.679397 19390 tablet_replica.cc:439] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } } }
I20260927 21:01:55.679486 19390 tablet_replica.cc:442] T 483f274661694e889060e9b1eebcfd61 P c106b6fa21cf42c2846c91019d191670: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:01:55.684265 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.681783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.16.190.194:37765)
0927 21:01:55.681825 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.681827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.681874 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:01:55.681962 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.681962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.681972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.682045 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:55.682196 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.682771 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.683554 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.683765 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.683846 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.684016 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:55.684050 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:55.684101 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.684164 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.684185 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:01:55.688989 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.686792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:33693 (local address 127.16.190.194:37765)
0927 21:01:55.686830 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:55.686831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:55.686851 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:01:55.686907 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:55.686907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:55.686915 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:55.686952 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:55.687080 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.687530 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.688263 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.688446 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.688528 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.688710 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:55.688745 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:55.688805 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:55.688890 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 21:01:55.688906 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:55.688989 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:55.686727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37765 (local address 127.16.190.194:33693)
0927 21:01:55.686802 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:55.686815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:55.686848 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:55.686961 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:55.686962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:55.686963 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:55.687013 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.687015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.687610 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.687611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.688180 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:55.688181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:55.688528 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:55.688529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:55.688589 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:55.688604 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:55.688805 (+ 201us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:55.688806 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:01:55.688886 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:01:55.688923 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:01:55.691511 19034 raft_consensus.cc:1261] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.691514 19312 raft_consensus.cc:1261] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.691720 19391 consensus_queue.cc:1035] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:55.691821 19393 consensus_queue.cc:1035] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:55.694439 19410 mvcc.cc:205] Tried to move back new op lower bound from 7334063782671134720 to 7334063781939093504. Current Snapshot: MvccSnapshot[applied={T|T < 7334063782671134720}]
I20260927 21:01:55.706157 18876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:55.706284 18876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:55.707157 18876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Sending DeleteTablet for 3 replicas of tablet 8286f3eb9a43420fbe70069ef10630ee
I20260927 21:01:55.707482 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 8286f3eb9a43420fbe70069ef10630ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:55 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260927 21:01:55.707598 19014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8286f3eb9a43420fbe70069ef10630ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:55 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56082
I20260927 21:01:55.707751 19424 tablet_replica.cc:330] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: stopping tablet replica
I20260927 21:01:55.707868 19424 raft_consensus.cc:2229] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:55.707854 19154 tablet_service.cc:1514] Processing DeleteTablet for tablet 8286f3eb9a43420fbe70069ef10630ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:55 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35042
I20260927 21:01:55.707821 18876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 21:01:55.707930 19424 raft_consensus.cc:2258] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 21:01:55.707960 18876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:55.708261 19425 tablet_replica.cc:330] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: stopping tablet replica
I20260927 21:01:55.708335 19426 tablet_replica.cc:330] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: stopping tablet replica
I20260927 21:01:55.708444 19426 raft_consensus.cc:2229] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:55.708549 19426 raft_consensus.cc:2258] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:55.708545 19425 raft_consensus.cc:2229] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:55.708663 19425 raft_consensus.cc:2258] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:55.709208 19425 ts_tablet_manager.cc:1905] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:55.709457 19426 ts_tablet_manager.cc:1905] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:55.709465 19014 tablet_service.cc:1467] Processing CreateTablet for tablet 04f7d709823b481f99e528cc2ae425e0 (DEFAULT_TABLE table=test-table [id=5193941a33264ded8e02626c5d1b5569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.709554 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f7d709823b481f99e528cc2ae425e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.710806 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 04f7d709823b481f99e528cc2ae425e0 (DEFAULT_TABLE table=test-table [id=5193941a33264ded8e02626c5d1b5569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.710903 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f7d709823b481f99e528cc2ae425e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.711092 19425 ts_tablet_manager.cc:1918] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:55.711167 19425 log.cc:1196] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/8286f3eb9a43420fbe70069ef10630ee
I20260927 21:01:55.711436 19425 ts_tablet_manager.cc:1939] T 8286f3eb9a43420fbe70069ef10630ee P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting consensus metadata
I20260927 21:01:55.711606 19426 ts_tablet_manager.cc:1918] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:55.711722 19426 log.cc:1196] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/8286f3eb9a43420fbe70069ef10630ee
I20260927 21:01:55.712023 19426 ts_tablet_manager.cc:1939] T 8286f3eb9a43420fbe70069ef10630ee P 0fa43adda6064881ba08c1b2902130e2: Deleting consensus metadata
I20260927 21:01:55.712181 19386 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Bootstrap starting.
I20260927 21:01:55.712188 19383 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Bootstrap starting.
I20260927 21:01:55.712606 19386 tablet_bootstrap.cc:654] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.712666 19383 tablet_bootstrap.cc:654] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.713029 18864 catalog_manager.cc:4841] TS 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765): tablet 8286f3eb9a43420fbe70069ef10630ee (table test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]) successfully deleted
I20260927 21:01:55.713090 18863 catalog_manager.cc:4841] TS 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249): tablet 8286f3eb9a43420fbe70069ef10630ee (table test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]) successfully deleted
I20260927 21:01:55.713253 19386 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: No bootstrap required, opened a new log
I20260927 21:01:55.713302 19386 ts_tablet_manager.cc:1397] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:55.713397 19383 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: No bootstrap required, opened a new log
I20260927 21:01:55.713429 19383 ts_tablet_manager.cc:1397] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 21:01:55.713465 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 04f7d709823b481f99e528cc2ae425e0 (DEFAULT_TABLE table=test-table [id=5193941a33264ded8e02626c5d1b5569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:55.713446 19386 raft_consensus.cc:348] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.713518 19386 raft_consensus.cc:374] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.713538 19386 raft_consensus.cc:729] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.713574 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04f7d709823b481f99e528cc2ae425e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:55.713555 19383 raft_consensus.cc:348] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.713577 19386 consensus_queue.cc:260] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.713613 19383 raft_consensus.cc:374] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.713636 19383 raft_consensus.cc:729] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9b9cae411b4ec5954b8797e4b9869b, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.713662 19386 ts_tablet_manager.cc:1428] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.713691 19383 consensus_queue.cc:260] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.713773 19383 ts_tablet_manager.cc:1428] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.714824 19385 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Bootstrap starting.
I20260927 21:01:55.715142 19385 tablet_bootstrap.cc:654] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:55.715670 19385 tablet_bootstrap.cc:492] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: No bootstrap required, opened a new log
I20260927 21:01:55.715737 19385 ts_tablet_manager.cc:1397] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:55.715914 19385 raft_consensus.cc:348] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.715987 19385 raft_consensus.cc:374] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:55.715960 19421 raft_consensus.cc:482] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:55.716015 19385 raft_consensus.cc:729] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Initialized, Role: FOLLOWER
I20260927 21:01:55.716022 19421 raft_consensus.cc:504] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.716055 19385 consensus_queue.cc:260] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.716142 19385 ts_tablet_manager.cc:1428] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:55.716148 19421 leader_election.cc:288] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:55.716310 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f7d709823b481f99e528cc2ae425e0" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" is_pre_election: true
I20260927 21:01:55.716313 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f7d709823b481f99e528cc2ae425e0" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670" is_pre_election: true
I20260927 21:01:55.716372 19312 raft_consensus.cc:2454] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.716372 19034 raft_consensus.cc:2454] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:55.716488 19107 leader_election.cc:302] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9b9cae411b4ec5954b8797e4b9869b, 0fa43adda6064881ba08c1b2902130e2; no voters:
I20260927 21:01:55.716567 19421 raft_consensus.cc:2781] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:55.716594 19421 raft_consensus.cc:482] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:55.716611 19421 raft_consensus.cc:3037] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.715718 19424 ts_tablet_manager.cc:1905] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:55.717229 19421 raft_consensus.cc:504] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.717391 19421 leader_election.cc:288] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:55.717571 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f7d709823b481f99e528cc2ae425e0" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670"
I20260927 21:01:55.717638 19312 raft_consensus.cc:3037] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.717681 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04f7d709823b481f99e528cc2ae425e0" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
I20260927 21:01:55.717741 19034 raft_consensus.cc:3037] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:55.718305 19312 raft_consensus.cc:2454] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.718425 19034 raft_consensus.cc:2454] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:55.718480 19106 leader_election.cc:302] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:55.718571 19424 ts_tablet_manager.cc:1918] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:55.718588 19421 raft_consensus.cc:2781] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:55.718640 19424 log.cc:1196] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/8286f3eb9a43420fbe70069ef10630ee
I20260927 21:01:55.718626 19421 raft_consensus.cc:686] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Running, Role: LEADER
I20260927 21:01:55.718690 19421 consensus_queue.cc:237] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:55.718890 19424 ts_tablet_manager.cc:1939] T 8286f3eb9a43420fbe70069ef10630ee P c106b6fa21cf42c2846c91019d191670: Deleting consensus metadata
I20260927 21:01:55.719195 18876 catalog_manager.cc:5495] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } health_report { overall_health: UNKNOWN } } }
I20260927 21:01:55.719297 18862 catalog_manager.cc:4841] TS c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867): tablet 8286f3eb9a43420fbe70069ef10630ee (table test-table [id=403c1d68dd2046acac0b1be8fe3d99d7]) successfully deleted
I20260927 21:01:55.727010 19312 raft_consensus.cc:1261] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.727046 19034 raft_consensus.cc:1261] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:55.727224 19421 consensus_queue.cc:1035] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:55.727358 19422 consensus_queue.cc:1035] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:01:55.731741 19408 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:01:57.249640 18877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:57.249743 18877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:57.250638 18877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Sending DeleteTablet for 3 replicas of tablet 04f7d709823b481f99e528cc2ae425e0
I20260927 21:01:57.250923 19014 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f7d709823b481f99e528cc2ae425e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56082
I20260927 21:01:57.250962 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f7d709823b481f99e528cc2ae425e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260927 21:01:57.250959 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 04f7d709823b481f99e528cc2ae425e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35042
I20260927 21:01:57.251196 19454 tablet_replica.cc:330] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: stopping tablet replica
I20260927 21:01:57.251264 19454 raft_consensus.cc:2229] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:57.251330 19454 raft_consensus.cc:2258] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.251559 19454 ts_tablet_manager.cc:1905] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.253108 19454 ts_tablet_manager.cc:1918] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:01:57.253218 19454 log.cc:1196] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/04f7d709823b481f99e528cc2ae425e0
I20260927 21:01:57.253557 19454 ts_tablet_manager.cc:1939] T 04f7d709823b481f99e528cc2ae425e0 P 0fa43adda6064881ba08c1b2902130e2: Deleting consensus metadata
I20260927 21:01:57.253886 18864 catalog_manager.cc:4841] TS 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765): tablet 04f7d709823b481f99e528cc2ae425e0 (table test-table [id=5193941a33264ded8e02626c5d1b5569]) successfully deleted
I20260927 21:01:57.255825 19452 tablet_replica.cc:330] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: stopping tablet replica
I20260927 21:01:57.255885 19452 raft_consensus.cc:2229] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:57.255916 19452 raft_consensus.cc:2258] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.256050 19453 tablet_replica.cc:330] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: stopping tablet replica
I20260927 21:01:57.256165 19453 raft_consensus.cc:2229] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:57.256206 19452 ts_tablet_manager.cc:1905] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.256222 19453 raft_consensus.cc:2258] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.256469 19453 ts_tablet_manager.cc:1905] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.257905 19452 ts_tablet_manager.cc:1918] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:01:57.257978 19452 log.cc:1196] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/04f7d709823b481f99e528cc2ae425e0
I20260927 21:01:57.258167 19453 ts_tablet_manager.cc:1918] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:01:57.258231 19453 log.cc:1196] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/04f7d709823b481f99e528cc2ae425e0
I20260927 21:01:57.258230 19452 ts_tablet_manager.cc:1939] T 04f7d709823b481f99e528cc2ae425e0 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting consensus metadata
I20260927 21:01:57.258435 19453 ts_tablet_manager.cc:1939] T 04f7d709823b481f99e528cc2ae425e0 P c106b6fa21cf42c2846c91019d191670: Deleting consensus metadata
I20260927 21:01:57.258611 18863 catalog_manager.cc:4841] TS 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249): tablet 04f7d709823b481f99e528cc2ae425e0 (table test-table [id=5193941a33264ded8e02626c5d1b5569]) successfully deleted
I20260927 21:01:57.259001 18862 catalog_manager.cc:4841] TS c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867): tablet 04f7d709823b481f99e528cc2ae425e0 (table test-table [id=5193941a33264ded8e02626c5d1b5569]) successfully deleted
Sep 27 21:01:57 dist-test-slave-6c7s krb5kdc[18832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:01:57 dist-test-slave-6c7s krb5kdc[18832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:01:57.281669 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.268347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.16.190.254:37653)
0927 21:01:57.268519 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.268521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.268568 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:01:57.268699 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.268700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.268713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.268779 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:01:57.269398 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.269923 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.270701 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.270923 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.271011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.278811 (+ 7800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:01:57.278814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:01:57.278815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:01:57.278833 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:01:57.280362 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:57.280748 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:57.280749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:57.280750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:57.280863 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:01:57.281080 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:01:57.281081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:01:57.281082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:01:57.281213 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:01:57.281319 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.281461 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.281554 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260927 21:01:57.283442 18877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:57.283601 18877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:57.285188 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 (DEFAULT_TABLE table=test-table [id=f523112545ad4d21bd59dd309569c156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.285188 19153 tablet_service.cc:1467] Processing CreateTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 (DEFAULT_TABLE table=test-table [id=f523112545ad4d21bd59dd309569c156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.285193 19014 tablet_service.cc:1467] Processing CreateTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 (DEFAULT_TABLE table=test-table [id=f523112545ad4d21bd59dd309569c156]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.285337 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6437b82c08b4403ba0b3fe3533a6d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.285336 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6437b82c08b4403ba0b3fe3533a6d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.285362 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6437b82c08b4403ba0b3fe3533a6d781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.286469 19463 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Bootstrap starting.
I20260927 21:01:57.286864 19464 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Bootstrap starting.
I20260927 21:01:57.286870 19463 tablet_bootstrap.cc:654] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.286942 19465 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Bootstrap starting.
I20260927 21:01:57.287319 19464 tablet_bootstrap.cc:654] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.287319 19465 tablet_bootstrap.cc:654] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.287966 19465 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: No bootstrap required, opened a new log
I20260927 21:01:57.287958 19464 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: No bootstrap required, opened a new log
I20260927 21:01:57.287994 19463 tablet_bootstrap.cc:492] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: No bootstrap required, opened a new log
I20260927 21:01:57.288025 19464 ts_tablet_manager.cc:1397] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:57.288064 19465 ts_tablet_manager.cc:1397] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:57.288103 19463 ts_tablet_manager.cc:1397] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:57.288178 19464 raft_consensus.cc:348] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288210 19463 raft_consensus.cc:348] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288210 19465 raft_consensus.cc:348] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288275 19464 raft_consensus.cc:374] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.288278 19465 raft_consensus.cc:374] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.288278 19463 raft_consensus.cc:374] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.288309 19464 raft_consensus.cc:729] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.288311 19463 raft_consensus.cc:729] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9b9cae411b4ec5954b8797e4b9869b, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.288311 19465 raft_consensus.cc:729] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.288355 19464 consensus_queue.cc:260] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288358 19465 consensus_queue.cc:260] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288374 19463 consensus_queue.cc:260] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.288462 19465 ts_tablet_manager.cc:1428] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:57.288518 19464 ts_tablet_manager.cc:1428] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:57.288540 19463 ts_tablet_manager.cc:1428] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:57.344866 19435 raft_consensus.cc:482] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:57.344985 19435 raft_consensus.cc:504] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.345477 19435 leader_election.cc:288] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:57.345755 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6437b82c08b4403ba0b3fe3533a6d781" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670" is_pre_election: true
I20260927 21:01:57.345875 19312 raft_consensus.cc:2454] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:57.345803 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6437b82c08b4403ba0b3fe3533a6d781" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" is_pre_election: true
I20260927 21:01:57.345974 19034 raft_consensus.cc:2454] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 0.
I20260927 21:01:57.346235 19107 leader_election.cc:302] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9b9cae411b4ec5954b8797e4b9869b, 0fa43adda6064881ba08c1b2902130e2; no voters:
I20260927 21:01:57.346411 19435 raft_consensus.cc:2781] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:57.346455 19435 raft_consensus.cc:482] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:57.346479 19435 raft_consensus.cc:3037] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.347680 19435 raft_consensus.cc:504] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.347851 19435 leader_election.cc:288] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867)
I20260927 21:01:57.348287 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6437b82c08b4403ba0b3fe3533a6d781" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c106b6fa21cf42c2846c91019d191670"
I20260927 21:01:57.348315 19034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6437b82c08b4403ba0b3fe3533a6d781" candidate_uuid: "0fa43adda6064881ba08c1b2902130e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
I20260927 21:01:57.348402 19312 raft_consensus.cc:3037] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.348409 19034 raft_consensus.cc:3037] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.349334 19312 raft_consensus.cc:2454] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:57.349491 19034 raft_consensus.cc:2454] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fa43adda6064881ba08c1b2902130e2 in term 1.
I20260927 21:01:57.349527 19106 leader_election.cc:302] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:57.349632 19435 raft_consensus.cc:2781] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:57.349684 19435 raft_consensus.cc:686] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Running, Role: LEADER
I20260927 21:01:57.349730 19435 consensus_queue.cc:237] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.350411 18876 catalog_manager.cc:5495] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "0fa43adda6064881ba08c1b2902130e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } health_report { overall_health: UNKNOWN } } }
I20260927 21:01:57.366815 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.364027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.16.190.194:37765)
0927 21:01:57.364203 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.364206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.364216 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:01:57.364241 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.364242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.364256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.364522 (+ 266us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:57.364553 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.365112 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.365964 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.366239 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.366310 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.366507 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.366547 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.366601 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.366681 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.366706 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260927 21:01:57.369391 19034 raft_consensus.cc:1261] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:57.369460 19312 raft_consensus.cc:1261] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Refusing update from remote peer 0fa43adda6064881ba08c1b2902130e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:57.369601 19435 consensus_queue.cc:1035] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:57.369796 19441 consensus_queue.cc:1035] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:57.379786 18877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:57.379899 18877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:57.381057 18877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Sending DeleteTablet for 3 replicas of tablet 6437b82c08b4403ba0b3fe3533a6d781
I20260927 21:01:57.381304 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260927 21:01:57.381340 19014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56082
I20260927 21:01:57.381446 19452 tablet_replica.cc:330] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: stopping tablet replica
I20260927 21:01:57.381453 19453 tablet_replica.cc:330] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: stopping tablet replica
I20260927 21:01:57.381511 19453 raft_consensus.cc:2229] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:57.381507 19452 raft_consensus.cc:2229] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:57.381582 19452 raft_consensus.cc:2258] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.381579 19453 raft_consensus.cc:2258] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.381846 19452 ts_tablet_manager.cc:1905] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.381850 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet 6437b82c08b4403ba0b3fe3533a6d781 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:57 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35042
I20260927 21:01:57.382176 18877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:01:57.382337 18877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:01:57.383487 19454 tablet_replica.cc:330] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: stopping tablet replica
I20260927 21:01:57.383567 19454 raft_consensus.cc:2229] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:57.383635 19454 raft_consensus.cc:2258] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:57.383715 19294 tablet_service.cc:1467] Processing CreateTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 (DEFAULT_TABLE table=test-table [id=6282518245084528abc7e06bbc7933a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.383778 19014 tablet_service.cc:1467] Processing CreateTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 (DEFAULT_TABLE table=test-table [id=6282518245084528abc7e06bbc7933a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.383834 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeb069b484e49758b9492a1c57aa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.383951 19454 ts_tablet_manager.cc:1905] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.384094 19153 tablet_service.cc:1467] Processing CreateTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 (DEFAULT_TABLE table=test-table [id=6282518245084528abc7e06bbc7933a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:01:57.384202 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeb069b484e49758b9492a1c57aa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.385279 19464 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Bootstrap starting.
I20260927 21:01:57.385538 19454 ts_tablet_manager.cc:1918] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:57.385592 19454 log.cc:1196] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/6437b82c08b4403ba0b3fe3533a6d781
I20260927 21:01:57.385591 19464 tablet_bootstrap.cc:654] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.385809 19454 ts_tablet_manager.cc:1939] T 6437b82c08b4403ba0b3fe3533a6d781 P 0fa43adda6064881ba08c1b2902130e2: Deleting consensus metadata
I20260927 21:01:57.385993 19465 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Bootstrap starting.
I20260927 21:01:57.386278 19465 tablet_bootstrap.cc:654] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.386584 19452 ts_tablet_manager.cc:1918] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:57.386656 19452 log.cc:1196] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/6437b82c08b4403ba0b3fe3533a6d781
I20260927 21:01:57.386863 19465 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: No bootstrap required, opened a new log
I20260927 21:01:57.386910 19465 ts_tablet_manager.cc:1397] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:01:57.386914 19452 ts_tablet_manager.cc:1939] T 6437b82c08b4403ba0b3fe3533a6d781 P 0b9b9cae411b4ec5954b8797e4b9869b: Deleting consensus metadata
I20260927 21:01:57.387065 19014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceeb069b484e49758b9492a1c57aa3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:01:57.387019 19465 raft_consensus.cc:348] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.387102 19465 raft_consensus.cc:374] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.387135 19465 raft_consensus.cc:729] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa43adda6064881ba08c1b2902130e2, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.387174 19465 consensus_queue.cc:260] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.387270 19465 ts_tablet_manager.cc:1428] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:01:57.388476 19463 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b: Bootstrap starting.
I20260927 21:01:57.388698 18863 catalog_manager.cc:4841] TS 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249): tablet 6437b82c08b4403ba0b3fe3533a6d781 (table test-table [id=f523112545ad4d21bd59dd309569c156]) successfully deleted
I20260927 21:01:57.388947 18864 catalog_manager.cc:4841] TS 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765): tablet 6437b82c08b4403ba0b3fe3533a6d781 (table test-table [id=f523112545ad4d21bd59dd309569c156]) successfully deleted
I20260927 21:01:57.389184 19453 ts_tablet_manager.cc:1905] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:57.389282 19463 tablet_bootstrap.cc:654] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b: Neither blocks nor log segments found. Creating new log.
I20260927 21:01:57.389838 19464 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: No bootstrap required, opened a new log
I20260927 21:01:57.389895 19464 ts_tablet_manager.cc:1397] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:01:57.390048 19464 raft_consensus.cc:348] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.390131 19464 raft_consensus.cc:374] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.390158 19464 raft_consensus.cc:729] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.390208 19464 consensus_queue.cc:260] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.390307 19464 ts_tablet_manager.cc:1428] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:57.390730 19463 tablet_bootstrap.cc:492] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b: No bootstrap required, opened a new log
I20260927 21:01:57.390789 19463 ts_tablet_manager.cc:1397] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:01:57.390914 19453 ts_tablet_manager.cc:1918] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:01:57.390967 19453 log.cc:1196] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/6437b82c08b4403ba0b3fe3533a6d781
I20260927 21:01:57.390936 19463 raft_consensus.cc:348] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.391010 19463 raft_consensus.cc:374] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:01:57.391057 19463 raft_consensus.cc:729] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9b9cae411b4ec5954b8797e4b9869b, State: Initialized, Role: FOLLOWER
I20260927 21:01:57.391105 19463 consensus_queue.cc:260] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.391193 19453 ts_tablet_manager.cc:1939] T 6437b82c08b4403ba0b3fe3533a6d781 P c106b6fa21cf42c2846c91019d191670: Deleting consensus metadata
I20260927 21:01:57.391214 19463 ts_tablet_manager.cc:1428] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:01:57.391713 18862 catalog_manager.cc:4841] TS c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867): tablet 6437b82c08b4403ba0b3fe3533a6d781 (table test-table [id=f523112545ad4d21bd59dd309569c156]) successfully deleted
I20260927 21:01:57.505962 19469 raft_consensus.cc:482] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:01:57.506067 19469 raft_consensus.cc:504] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.506418 19469 leader_election.cc:288] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765)
I20260927 21:01:57.510844 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.506678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:58663 (local address 127.16.190.194:37765)
0927 21:01:57.506731 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.506733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.506865 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:01:57.506981 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.506982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.506995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.507068 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:57.507355 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.508108 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.509845 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.510423 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.510508 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.510517 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.510726 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.510767 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:01:57.510861 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.506681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:46381 (local address 127.16.190.193:42249)
0927 21:01:57.506824 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.506826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.506856 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:01:57.506978 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.506979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.506992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.507069 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:01:57.507351 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.508118 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.509846 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.510430 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.510508 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.510517 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.510729 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.510769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260927 21:01:57.510893 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.506607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37765 (local address 127.16.190.195:58663)
0927 21:01:57.506786 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:57.506798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:57.506874 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:57.507133 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:57.507134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:57.507137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:57.507263 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.507264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.508243 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.508244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.509745 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.509748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.510542 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.510543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.510632 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.510639 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:01:57.510705 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:01:57.510765 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":123,"thread_start_us":110,"threads_started":1}
I20260927 21:01:57.510890 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.506656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42249 (local address 127.16.190.195:46381)
0927 21:01:57.506801 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:57.506811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:57.506874 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:57.507133 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:57.507134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:57.507139 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:01:57.507258 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.507260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.508265 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.508266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.509745 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.509748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.510542 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.510543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.510632 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.510638 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:01:57.510705 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:01:57.510765 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":97,"thread_start_us":105,"threads_started":1}
I20260927 21:01:57.511317 19173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeb069b484e49758b9492a1c57aa3a3" candidate_uuid: "c106b6fa21cf42c2846c91019d191670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa43adda6064881ba08c1b2902130e2" is_pre_election: true
I20260927 21:01:57.511473 19173 raft_consensus.cc:2454] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106b6fa21cf42c2846c91019d191670 in term 0.
I20260927 21:01:57.511415 19033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeb069b484e49758b9492a1c57aa3a3" candidate_uuid: "c106b6fa21cf42c2846c91019d191670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" is_pre_election: true
I20260927 21:01:57.511525 19033 raft_consensus.cc:2454] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c106b6fa21cf42c2846c91019d191670 in term 0.
I20260927 21:01:57.511750 19247 leader_election.cc:302] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9b9cae411b4ec5954b8797e4b9869b, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:57.511902 19469 raft_consensus.cc:2781] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:01:57.511941 19469 raft_consensus.cc:482] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:01:57.511969 19469 raft_consensus.cc:3037] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.512673 19469 raft_consensus.cc:504] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.512859 19469 leader_election.cc:288] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [CANDIDATE]: Term 1 election: Requested vote from peers 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249), 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765)
I20260927 21:01:57.512975 19033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeb069b484e49758b9492a1c57aa3a3" candidate_uuid: "c106b6fa21cf42c2846c91019d191670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9b9cae411b4ec5954b8797e4b9869b"
I20260927 21:01:57.512981 19173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceeb069b484e49758b9492a1c57aa3a3" candidate_uuid: "c106b6fa21cf42c2846c91019d191670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa43adda6064881ba08c1b2902130e2"
I20260927 21:01:57.513055 19033 raft_consensus.cc:3037] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.513077 19173 raft_consensus.cc:3037] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:01:57.513758 19173 raft_consensus.cc:2454] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106b6fa21cf42c2846c91019d191670 in term 1.
I20260927 21:01:57.513789 19033 raft_consensus.cc:2454] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c106b6fa21cf42c2846c91019d191670 in term 1.
I20260927 21:01:57.513942 19248 leader_election.cc:302] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa43adda6064881ba08c1b2902130e2, c106b6fa21cf42c2846c91019d191670; no voters:
I20260927 21:01:57.514046 19469 raft_consensus.cc:2781] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:01:57.514202 19469 raft_consensus.cc:686] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 1 LEADER]: Becoming Leader. State: Replica: c106b6fa21cf42c2846c91019d191670, State: Running, Role: LEADER
I20260927 21:01:57.514266 19469 consensus_queue.cc:237] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } }
I20260927 21:01:57.515031 18877 catalog_manager.cc:5495] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 reported cstate change: term changed from 0 to 1, leader changed from <none> to c106b6fa21cf42c2846c91019d191670 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "c106b6fa21cf42c2846c91019d191670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c106b6fa21cf42c2846c91019d191670" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42867 } health_report { overall_health: HEALTHY } } }
I20260927 21:01:57.539803 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.536891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.16.190.195:42867)
0927 21:01:57.537100 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.537102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.537109 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:01:57.537155 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.537156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.537174 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.537269 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:57.537387 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.538042 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.538933 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.539173 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.539251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.539414 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.539554 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.539644 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.539667 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.539686 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":137,"threads_started":1}
I20260927 21:01:57.545056 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.542678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:59695 (local address 127.16.190.194:37765)
0927 21:01:57.542753 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.542755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.542885 (+ 130us) server_negotiation.cc:407] Connection header received
0927 21:01:57.542942 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.542943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.542954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.543004 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:57.543193 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.543643 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.544404 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.544586 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.544651 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.544843 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.544877 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.544917 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.544979 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.545000 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:01:57.545097 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.542567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37765 (local address 127.16.190.195:59695)
0927 21:01:57.542815 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:57.542826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:57.542885 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:57.543049 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:57.543050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:57.543065 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:57.543127 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.543128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.543729 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.543730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.544326 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.544327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.544681 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.544682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.544731 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.544744 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.544940 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.544941 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:01:57.544978 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 21:01:57.545015 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":178,"threads_started":1}
I20260927 21:01:57.547494 19033 raft_consensus.cc:1261] T ceeb069b484e49758b9492a1c57aa3a3 P 0b9b9cae411b4ec5954b8797e4b9869b [term 1 FOLLOWER]: Refusing update from remote peer c106b6fa21cf42c2846c91019d191670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:57.547566 19173 raft_consensus.cc:1261] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Refusing update from remote peer c106b6fa21cf42c2846c91019d191670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:01:57.547837 19483 consensus_queue.cc:1035] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9b9cae411b4ec5954b8797e4b9869b" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:57.547977 19469 consensus_queue.cc:1035] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa43adda6064881ba08c1b2902130e2" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:01:57.555944 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.553351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:44927 (local address 127.16.190.195:42867)
0927 21:01:57.553397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:01:57.553399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:01:57.553510 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:01:57.553587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:01:57.553588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:01:57.553600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:01:57.553659 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:01:57.553773 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.554307 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.555277 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.555586 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.555654 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.555777 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.555802 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.555831 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 21:01:57.555873 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 21:01:57.555889 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:01:57.555972 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:01:57.553283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:42867 (local address 127.16.190.194:44927)
0927 21:01:57.553448 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:57.553461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:01:57.553528 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:01:57.553662 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:01:57.553663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:01:57.553665 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:01:57.553723 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.553725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.554403 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.554405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.555184 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:01:57.555188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:01:57.555669 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:01:57.555670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:01:57.555714 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:01:57.555719 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:01:57.555838 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:01:57.555839 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:01:57.555865 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 21:01:57.555903 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":119,"threads_started":1}
W20260927 21:01:57.556367 19436 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 21:01:59.081050 18877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:59.081153 18877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:01:59.082044 18877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c4592054a004e979744b9cb50da8662: Sending DeleteTablet for 3 replicas of tablet ceeb069b484e49758b9492a1c57aa3a3
I20260927 21:01:59.082304 19014 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:59 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56082
I20260927 21:01:59.082336 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:59 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52614
I20260927 21:01:59.082530 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18949
I20260927 21:01:59.082587 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet ceeb069b484e49758b9492a1c57aa3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:01:59 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:35042
I20260927 21:01:59.084060 19519 tablet_replica.cc:330] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: stopping tablet replica
I20260927 21:01:59.084132 19519 raft_consensus.cc:2229] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:01:59.084193 19519 raft_consensus.cc:2258] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:59.084463 19519 ts_tablet_manager.cc:1905] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:59.084926 19517 tablet_replica.cc:330] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: stopping tablet replica
I20260927 21:01:59.085007 19517 raft_consensus.cc:2229] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:01:59.085124 19517 raft_consensus.cc:2258] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:01:59.085352 19517 ts_tablet_manager.cc:1905] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:01:59.086575 19519 ts_tablet_manager.cc:1918] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:01:59.086651 19519 log.cc:1196] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/ceeb069b484e49758b9492a1c57aa3a3
I20260927 21:01:59.086737 19517 ts_tablet_manager.cc:1918] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 21:01:59.086800 19517 log.cc:1196] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/ceeb069b484e49758b9492a1c57aa3a3
I20260927 21:01:59.086897 19519 ts_tablet_manager.cc:1939] T ceeb069b484e49758b9492a1c57aa3a3 P 0fa43adda6064881ba08c1b2902130e2: Deleting consensus metadata
I20260927 21:01:59.087103 19517 ts_tablet_manager.cc:1939] T ceeb069b484e49758b9492a1c57aa3a3 P c106b6fa21cf42c2846c91019d191670: Deleting consensus metadata
I20260927 21:01:59.087410 18864 catalog_manager.cc:4841] TS 0fa43adda6064881ba08c1b2902130e2 (127.16.190.194:37765): tablet ceeb069b484e49758b9492a1c57aa3a3 (table test-table [id=6282518245084528abc7e06bbc7933a9]) successfully deleted
I20260927 21:01:59.087603 18862 catalog_manager.cc:4841] TS c106b6fa21cf42c2846c91019d191670 (127.16.190.195:42867): tablet ceeb069b484e49758b9492a1c57aa3a3 (table test-table [id=6282518245084528abc7e06bbc7933a9]) successfully deleted
W20260927 21:01:59.088652 18863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:42249 (error 108)
W20260927 21:01:59.089326 19521 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:01:59.089089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42249 (local address 127.0.0.1:56084)
0927 21:01:59.089231 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:01:59.089285 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42249: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
W20260927 21:01:59.089440 18863 catalog_manager.cc:4584] TS 0b9b9cae411b4ec5954b8797e4b9869b (127.16.190.193:42249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ceeb069b484e49758b9492a1c57aa3a3: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42249: connect: Connection reset by peer (error 104)
I20260927 21:01:59.089648 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19089
I20260927 21:01:59.096166 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19229
I20260927 21:01:59.102052 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 18845
2026-09-27T21:01:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5563 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:01:59 dist-test-slave-6c7s krb5kdc[19524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:01:59 dist-test-slave-6c7s krb5kdc[19524](info): set up 2 sockets
Sep 27 21:01:59 dist-test-slave-6c7s krb5kdc[19524](info): commencing operation
krb5kdc: starting...
W20260927 21:02:00.141541 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:00.265344 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45279
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:00.347772 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:00.348021 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:00.348059 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:00.349291 19537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:00.349354 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:00.349377 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:00.349399 19537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:00.349438 19537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:00.350899 19537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:00.351177 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:00.351402 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:00.356359 19537 server_base.cc:1034] running on GCE node
W20260927 21:02:00.356305 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.356295 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.356302 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.356910 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:00.357194 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:00.358394 19537 hybrid_clock.cc:648] HybridClock initialized: now 1790542920358384 us; error 42 us; skew 500 ppm
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:00.360960 19537 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:00.361438 19537 webserver.cc:460] Webserver started at http://127.16.190.254:34801/ using document root <none> and password file <none>
I20260927 21:02:00.361689 19537 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:00.361735 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:00.361826 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:00.362864 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "8e4da0e074d943b0b21df6be82000725"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "65648463cd84e133bae5fd02e4cfa052"
server_key_iv: "72606064c3a8670736985a104dfb6b83"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.363202 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "8e4da0e074d943b0b21df6be82000725"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "65648463cd84e133bae5fd02e4cfa052"
server_key_iv: "72606064c3a8670736985a104dfb6b83"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.364585 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:00.365366 19552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.365540 19537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.365628 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "8e4da0e074d943b0b21df6be82000725"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "65648463cd84e133bae5fd02e4cfa052"
server_key_iv: "72606064c3a8670736985a104dfb6b83"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.365693 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:00.391811 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:00.394354 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:00.394474 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:00.398361 19537 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:45279
I20260927 21:02:00.398756 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:00.398831 19604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:45279 every 8 connection(s)
I20260927 21:02:00.399490 19605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:00.400314 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19537
I20260927 21:02:00.400389 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:00.400583 17147 external_mini_cluster.cc:1428] Setting key 4f4eae49e7aecb1990cfd728cee58a78
I20260927 21:02:00.402885 19605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725: Bootstrap starting.
I20260927 21:02:00.403524 19605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:00.403817 19605 log.cc:824] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:00.404534 19605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725: No bootstrap required, opened a new log
I20260927 21:02:00.406605 19605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } }
I20260927 21:02:00.406796 19605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:00.406833 19605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4da0e074d943b0b21df6be82000725, State: Initialized, Role: FOLLOWER
I20260927 21:02:00.406977 19605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } }
I20260927 21:02:00.407083 19605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:00.407119 19605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:00.407169 19605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:00.407735 19605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } }
I20260927 21:02:00.407824 19605 leader_election.cc:302] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e4da0e074d943b0b21df6be82000725; no voters:
I20260927 21:02:00.408084 19605 leader_election.cc:288] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:00.408107 19610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:00.408237 19610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4da0e074d943b0b21df6be82000725, State: Running, Role: LEADER
I20260927 21:02:00.408334 19610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } }
I20260927 21:02:00.408367 19605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:00.408694 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e4da0e074d943b0b21df6be82000725. Latest consensus state: current_term: 1 leader_uuid: "8e4da0e074d943b0b21df6be82000725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } } }
I20260927 21:02:00.408769 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:00.409138 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e4da0e074d943b0b21df6be82000725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e4da0e074d943b0b21df6be82000725" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45279 } } }
I20260927 21:02:00.409231 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:00.409736 19617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:00.410432 19617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:00.412027 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.401202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.16.190.254:45279)
0927 21:02:00.401371 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:00.401379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:00.401404 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:02:00.401814 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:00.401821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:00.401866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:00.401977 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:00.402855 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.403848 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.404626 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.405093 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.405234 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.407754 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:00.407758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:00.407759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:00.407770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:00.410054 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.410426 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.410428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.410429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.410587 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.410867 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.410868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.410868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.411146 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:00.411672 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:00.411749 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 21:02:00.411887 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260927 21:02:00.412175 19617 catalog_manager.cc:1347] Generated new cluster ID: fcabbe9649204ebeb89616a7ebd6720b
I20260927 21:02:00.412231 19617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:00.419169 19617 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:00.419899 19617 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:00.427332 19617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e4da0e074d943b0b21df6be82000725: Generated new TSK 0
I20260927 21:02:00.427560 19617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:00.456377 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:00.539557 19631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:00.539803 19631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:00.539844 19631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:00.541220 19631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:00.541286 19631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:00.541333 19631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:00.542819 19631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:00.543095 19631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:00.543339 19631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:00.548446 19637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.548429 19639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.548591 19631 server_base.cc:1034] running on GCE node
W20260927 21:02:00.548429 19636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.548978 19631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:00.549204 19631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:00.550341 19631 hybrid_clock.cc:648] HybridClock initialized: now 1790542920550332 us; error 33 us; skew 500 ppm
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:00.553135 19631 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:00.553594 19631 webserver.cc:460] Webserver started at http://127.16.190.193:35647/ using document root <none> and password file <none>
I20260927 21:02:00.553849 19631 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:00.553895 19631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:00.554050 19631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:00.555123 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "f06918065a4a4e6d835de025a682aab0"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "52a59576ee6cb625ef6f28cba30cdc7a"
server_key_iv: "91db0dfe9b99b027da940814edf824ba"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.555502 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "f06918065a4a4e6d835de025a682aab0"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "52a59576ee6cb625ef6f28cba30cdc7a"
server_key_iv: "91db0dfe9b99b027da940814edf824ba"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.556965 19631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:00.557864 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.558072 19631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:00.558154 19631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "f06918065a4a4e6d835de025a682aab0"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "52a59576ee6cb625ef6f28cba30cdc7a"
server_key_iv: "91db0dfe9b99b027da940814edf824ba"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.558208 19631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:00.570340 19631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:00.572925 19631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:00.573057 19631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:00.573261 19631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:00.573585 19631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:00.573619 19631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.573652 19631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:00.573700 19631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.579349 19631 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:43895
I20260927 21:02:00.579433 19759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:43895 every 8 connection(s)
I20260927 21:02:00.579782 19631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:00.581805 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19631
I20260927 21:02:00.581919 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:00.582115 17147 external_mini_cluster.cc:1428] Setting key 788fbf5cc4469c0fc54502e18926f650
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:00.590793 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.580626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:36795 (local address 127.16.190.254:45279)
0927 21:02:00.580712 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:00.580714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:00.580835 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:02:00.581389 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:00.581390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:00.581400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:00.581457 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:00.582765 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.583271 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.584779 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.585087 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.585156 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.587692 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:00.587693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:00.587694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:00.587702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:00.589274 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.589735 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.589736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.589737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.589797 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.590085 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.590086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.590087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.590210 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:00.590289 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:00.590592 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:02:00.590690 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:02:00.591042 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.580516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45279 (local address 127.16.190.193:36795)
0927 21:02:00.580691 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:00.580743 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:00.581299 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:00.581534 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:00.581539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:00.582426 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:00.582687 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.582695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.583367 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.583368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.584532 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.584535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.585182 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.585183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.585336 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.585891 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:00.585901 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:00.587584 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:00.589383 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.589391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.589393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.589640 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.589868 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.589869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.589869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.590018 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.590281 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:00.590288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:00.590443 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:02:00.590553 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:02:00.590750 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260927 21:02:00.591516 19760 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45279
I20260927 21:02:00.591682 19760 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:00.591895 19760 heartbeater.cc:511] Master 127.16.190.254:45279 requested a full tablet report, sending...
I20260927 21:02:00.592607 19569 ts_manager.cc:194] Registered new tserver with Master: f06918065a4a4e6d835de025a682aab0 (127.16.190.193:43895)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:02:00.593372 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:36795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:00.612740 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:00.695493 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:00.695691 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:00.695720 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:00.696802 19765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:00.696852 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:00.696877 19765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:00.698334 19765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:00.698527 19765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:00.698704 19765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:00.704039 19765 server_base.cc:1034] running on GCE node
W20260927 21:02:00.704023 19771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.704023 19770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.704023 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.704493 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:00.704799 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:00.706014 19765 hybrid_clock.cc:648] HybridClock initialized: now 1790542920705992 us; error 38 us; skew 500 ppm
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:00.708634 19765 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:00.709272 19765 webserver.cc:460] Webserver started at http://127.16.190.194:36517/ using document root <none> and password file <none>
I20260927 21:02:00.709532 19765 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:00.709606 19765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:00.709801 19765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:00.710906 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "6b06302561ef4454a886190cbca09909"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "d096767478dfc20fc501347732902f18"
server_key_iv: "de55c814e6e6992a5c74ce9d33510ff1"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.711336 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "6b06302561ef4454a886190cbca09909"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "d096767478dfc20fc501347732902f18"
server_key_iv: "de55c814e6e6992a5c74ce9d33510ff1"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.713189 19765 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 21:02:00.714051 19780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.714253 19765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:00.714355 19765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "6b06302561ef4454a886190cbca09909"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "d096767478dfc20fc501347732902f18"
server_key_iv: "de55c814e6e6992a5c74ce9d33510ff1"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.714439 19765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:00.730069 19765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:00.732570 19765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:00.732892 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:00.733173 19765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:00.733498 19765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:00.733552 19765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.733606 19765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:00.733634 19765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.739703 19765 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:40985
I20260927 21:02:00.739778 19893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:40985 every 8 connection(s)
I20260927 21:02:00.740118 19765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:00.748602 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19765
I20260927 21:02:00.748714 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:00.748970 17147 external_mini_cluster.cc:1428] Setting key fabc5c5e52f5e825ef2b1e5d18ba0532
I20260927 21:02:00.750099 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.740942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43733 (local address 127.16.190.254:45279)
0927 21:02:00.740991 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:00.740993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:00.741167 (+ 174us) server_negotiation.cc:407] Connection header received
0927 21:02:00.741673 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:00.741674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:00.741684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:00.741750 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:00.742689 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.743196 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.744260 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.744453 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.744530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.746653 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:00.746655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:00.746655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:00.746662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:00.748339 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.748997 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.748999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.748999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.749071 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.749367 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.749368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.749369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.749505 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:00.749588 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:00.749961 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 21:02:00.750033 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:00.750474 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.740790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45279 (local address 127.16.190.194:43733)
0927 21:02:00.741009 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:00.741053 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:00.741569 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:00.741818 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:00.741822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:00.742395 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:00.742592 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.742596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.743287 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.743288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.744138 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.744141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.744534 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.744535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.744641 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.745173 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:00.745181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:00.746557 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:00.748488 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.748495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.748497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.748918 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.749146 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.749147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.749147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.749273 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.749591 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:00.749599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:00.749791 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 21:02:00.749919 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:02:00.750146 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":115,"threads_started":1}
I20260927 21:02:00.750968 19894 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45279
I20260927 21:02:00.751101 19894 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:00.751302 19894 heartbeater.cc:511] Master 127.16.190.254:45279 requested a full tablet report, sending...
I20260927 21:02:00.751711 19569 ts_manager.cc:194] Registered new tserver with Master: 6b06302561ef4454a886190cbca09909 (127.16.190.194:40985)
I20260927 21:02:00.752111 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:43733
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:00.779755 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:00.864478 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:00.864720 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:00.864745 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:00.865988 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:00.866044 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:00.866072 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:00.867491 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:00.867753 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:00.867990 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:00.873219 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:00.873261 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.873447 19899 server_base.cc:1034] running on GCE node
W20260927 21:02:00.873479 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:00.873735 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:00.874003 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:00.875205 19899 hybrid_clock.cc:648] HybridClock initialized: now 1790542920875190 us; error 43 us; skew 500 ppm
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:00.877944 19899 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:00.878450 19899 webserver.cc:460] Webserver started at http://127.16.190.195:37801/ using document root <none> and password file <none>
I20260927 21:02:00.878643 19899 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:00.878691 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:00.878784 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:00.879801 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "003d3a5e346e4e45996ae80458864444"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "bafb1aefe32caabc45944db79fe54110"
server_key_iv: "94b3c39f46f9d06739ff7faade1eaffc"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.880132 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "003d3a5e346e4e45996ae80458864444"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "bafb1aefe32caabc45944db79fe54110"
server_key_iv: "94b3c39f46f9d06739ff7faade1eaffc"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.881537 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:02:00.882357 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.882599 19899 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:00.882689 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "003d3a5e346e4e45996ae80458864444"
format_stamp: "Formatted at 2026-09-27 21:02:00 on dist-test-slave-6c7s"
server_key: "bafb1aefe32caabc45944db79fe54110"
server_key_iv: "94b3c39f46f9d06739ff7faade1eaffc"
server_key_version: "encryptionkey@0"
I20260927 21:02:00.882751 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:00.903312 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:00.905581 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:00.905702 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:00.905947 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:00.906255 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:00.906286 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.906309 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:00.906322 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:00.913069 19899 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34929
I20260927 21:02:00.913110 20027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34929 every 8 connection(s)
I20260927 21:02:00.913455 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:00.915308 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 19899
I20260927 21:02:00.915385 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:00.915607 17147 external_mini_cluster.cc:1428] Setting key 90d130c5c90680966fbe679db5cf6b3a
Sep 27 21:02:00 dist-test-slave-6c7s krb5kdc[19524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542920, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:00.923250 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.914156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:40905 (local address 127.16.190.254:45279)
0927 21:02:00.914222 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:00.914225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:00.914428 (+ 203us) server_negotiation.cc:407] Connection header received
0927 21:02:00.914956 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:00.914957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:00.914968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:00.915023 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:00.916144 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.916662 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.917791 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.918004 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.918094 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.920278 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:00.920280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:00.920280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:00.920289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:00.921766 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.922206 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.922208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.922208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.922275 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:00.922551 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:00.922552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:00.922552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:00.922682 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:00.922762 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:00.923065 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:02:00.923150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:02:00.923455 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:00.914126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45279 (local address 127.16.190.195:40905)
0927 21:02:00.914297 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:00.914337 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:00.914850 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:00.915106 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:00.915110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:00.915822 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:00.916036 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.916042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.916772 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.916773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.917681 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:00.917683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:00.918110 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:00.918111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:00.918240 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:00.918773 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:00.918782 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:00.920171 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:00.921866 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.921874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.921876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.922126 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.922356 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:00.922356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:00.922357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:00.922475 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:00.922753 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:00.922759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:00.922915 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 21:02:00.923029 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:02:00.923208 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260927 21:02:00.924052 20028 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45279
I20260927 21:02:00.924155 20028 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:00.924322 20028 heartbeater.cc:511] Master 127.16.190.254:45279 requested a full tablet report, sending...
I20260927 21:02:00.924754 19569 ts_manager.cc:194] Registered new tserver with Master: 003d3a5e346e4e45996ae80458864444 (127.16.190.195:34929)
I20260927 21:02:00.925189 19569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:40905
I20260927 21:02:00.927292 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:02:00.939076 19608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:00.937983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.16.190.254:45279)
0927 21:02:00.938074 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:00.938077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:00.938196 (+ 119us) server_negotiation.cc:407] Connection header received
0927 21:02:00.938298 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:00.938299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:00.938310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:00.938401 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:00.939015 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50982: BlockingRecv error: recv got EOF from 127.0.0.1:50982 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 21:02:00.940121 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19631
I20260927 21:02:00.945612 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19765
I20260927 21:02:00.950417 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19899
I20260927 21:02:00.954598 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 19537
2026-09-27T21:02:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1851 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:01 dist-test-slave-6c7s krb5kdc[20040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:01 dist-test-slave-6c7s krb5kdc[20040](info): set up 2 sockets
Sep 27 21:02:01 dist-test-slave-6c7s krb5kdc[20040](info): commencing operation
krb5kdc: starting...
W20260927 21:02:01.997526 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:02.133088 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:44989
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:02.216324 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:02.216591 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:02.216629 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:02.217895 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:02.217970 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:02.218006 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:02.218030 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:02.218082 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:02.219672 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:44989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:02.219959 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:02.220198 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:02.225097 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:02.225121 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:02.225128 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:02.225369 20053 server_base.cc:1034] running on GCE node
I20260927 21:02:02.225641 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:02.225890 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:02.227089 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790542922227068 us; error 44 us; skew 500 ppm
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542922, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:02.229835 20053 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:02.230343 20053 webserver.cc:460] Webserver started at http://127.16.190.254:34255/ using document root <none> and password file <none>
I20260927 21:02:02.230620 20053 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:02.230700 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:02.230872 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:02.232040 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "d8cf654a1f2546839abeb9addb63afa8"
format_stamp: "Formatted at 2026-09-27 21:02:02 on dist-test-slave-6c7s"
server_key: "94b94777ed5f3140aaaade5558825c21"
server_key_iv: "77c9c13b1b79d5ed0336eab43957ed78"
server_key_version: "encryptionkey@0"
I20260927 21:02:02.232442 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "d8cf654a1f2546839abeb9addb63afa8"
format_stamp: "Formatted at 2026-09-27 21:02:02 on dist-test-slave-6c7s"
server_key: "94b94777ed5f3140aaaade5558825c21"
server_key_iv: "77c9c13b1b79d5ed0336eab43957ed78"
server_key_version: "encryptionkey@0"
I20260927 21:02:02.234068 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:02.234968 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:02.235252 20053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:02.235363 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "d8cf654a1f2546839abeb9addb63afa8"
format_stamp: "Formatted at 2026-09-27 21:02:02 on dist-test-slave-6c7s"
server_key: "94b94777ed5f3140aaaade5558825c21"
server_key_iv: "77c9c13b1b79d5ed0336eab43957ed78"
server_key_version: "encryptionkey@0"
I20260927 21:02:02.235464 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:02.251539 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:02.254405 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:02.254701 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:02.258832 20053 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:44989
I20260927 21:02:02.258882 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:44989 every 8 connection(s)
I20260927 21:02:02.259413 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:02.260110 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:02.262393 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8: Bootstrap starting.
I20260927 21:02:02.262964 20121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:02.263216 20121 log.cc:824] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:02.263904 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8: No bootstrap required, opened a new log
I20260927 21:02:02.265622 20121 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } }
I20260927 21:02:02.265779 20121 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:02.265802 20121 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cf654a1f2546839abeb9addb63afa8, State: Initialized, Role: FOLLOWER
I20260927 21:02:02.265955 20121 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } }
I20260927 21:02:02.266040 20121 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:02.266063 20121 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:02.266091 20121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:02.266956 20121 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } }
I20260927 21:02:02.267040 20121 leader_election.cc:302] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8cf654a1f2546839abeb9addb63afa8; no voters:
I20260927 21:02:02.267208 20121 leader_election.cc:288] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:02.267256 20124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:02.267382 20124 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [term 1 LEADER]: Becoming Leader. State: Replica: d8cf654a1f2546839abeb9addb63afa8, State: Running, Role: LEADER
I20260927 21:02:02.267475 20121 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:02.267462 20124 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } }
I20260927 21:02:02.267995 20126 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8cf654a1f2546839abeb9addb63afa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } } }
I20260927 21:02:02.268007 20125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8cf654a1f2546839abeb9addb63afa8. Latest consensus state: current_term: 1 leader_uuid: "d8cf654a1f2546839abeb9addb63afa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8cf654a1f2546839abeb9addb63afa8" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44989 } } }
I20260927 21:02:02.268148 20125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:02.268139 20126 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:02.268383 20133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:02.268473 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20053
I20260927 21:02:02.268603 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:02.268848 17147 external_mini_cluster.cc:1428] Setting key be936d5dc7751b6a8080f47f72a8760b
I20260927 21:02:02.269126 20133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:02.271133 20133 catalog_manager.cc:1347] Generated new cluster ID: 26c402330a264d5fafc1dc4a739a6197
I20260927 21:02:02.271210 20133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:02.282400 20133 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:02.283123 20133 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:02.290721 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:02.269398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.16.190.254:44989)
0927 21:02:02.269588 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:02.269599 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:02.269837 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:02:02.269986 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:02.269993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:02.270069 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:02.270215 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:02.272993 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:02.274542 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:02.275646 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:02.276344 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:02.276472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:02.286018 (+ 9546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:02.286025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:02.286025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:02.286042 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:02.288373 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:02.288919 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:02.288923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:02.288924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:02.289093 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:02.289340 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:02.289341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:02.289342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:02.289668 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:02.290019 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:02.290115 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 21:02:02.290289 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260927 21:02:02.291889 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:02:02.295835 20133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8cf654a1f2546839abeb9addb63afa8: Generated new TSK 0
I20260927 21:02:02.296067 20133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:02.307534 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:02.303387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.16.190.254:44989)
0927 21:02:02.303453 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:02.303455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:02.303565 (+ 110us) server_negotiation.cc:407] Connection header received
0927 21:02:02.303642 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:02.303643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:02.303656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:02.303715 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:02.304079 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:02.304937 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:02.305773 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:02.306117 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:02.306201 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:02.306963 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:02.306964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:02.306965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:02.307243 (+ 278us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:02.307264 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:02.307272 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:02.307336 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:02.307410 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:02:02.307441 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:02:02.310544 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20053
2026-09-27T21:02:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1354 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20154](info): set up 2 sockets
Sep 27 21:02:02 dist-test-slave-6c7s krb5kdc[20154](info): commencing operation
krb5kdc: starting...
W20260927 21:02:03.373486 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:03.500032 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36895
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:03.581415 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:03.581676 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:03.581715 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:03.582916 20167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:03.582988 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:03.583014 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:03.583039 20167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:03.583071 20167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:03.584592 20167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:03.584919 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:03.585165 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:03.590672 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:03.590631 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:03.590631 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:03.590765 20167 server_base.cc:1034] running on GCE node
I20260927 21:02:03.591116 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:03.591418 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:03.592557 20167 hybrid_clock.cc:648] HybridClock initialized: now 1790542923592529 us; error 45 us; skew 500 ppm
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:03.595067 20167 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:03.595532 20167 webserver.cc:460] Webserver started at http://127.16.190.254:44127/ using document root <none> and password file <none>
I20260927 21:02:03.595791 20167 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:03.595837 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:03.596005 20167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:03.597149 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "412d8ed63dbf47688fa929a3c290cf8d"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "28ee9e6726a5e086189f02b4f8d585cb"
server_key_iv: "059268ea7eaa95b133da2fca86acaed2"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.597524 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "412d8ed63dbf47688fa929a3c290cf8d"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "28ee9e6726a5e086189f02b4f8d585cb"
server_key_iv: "059268ea7eaa95b133da2fca86acaed2"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.598946 20167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:03.599854 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.600060 20167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:03.600155 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "412d8ed63dbf47688fa929a3c290cf8d"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "28ee9e6726a5e086189f02b4f8d585cb"
server_key_iv: "059268ea7eaa95b133da2fca86acaed2"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.600250 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:03.614087 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:03.616185 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:03.616356 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:03.620591 20167 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36895
I20260927 21:02:03.620642 20234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36895 every 8 connection(s)
I20260927 21:02:03.621048 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:03.622205 20235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:03.624715 20235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d: Bootstrap starting.
I20260927 21:02:03.625380 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20167
I20260927 21:02:03.625550 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:03.625591 20235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:03.625782 17147 external_mini_cluster.cc:1428] Setting key 02c4b44d0c8fcaac32b5289ed2ffafe1
I20260927 21:02:03.625941 20235 log.cc:824] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:03.626819 20235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d: No bootstrap required, opened a new log
I20260927 21:02:03.629428 20235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } }
I20260927 21:02:03.629637 20235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:03.629676 20235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412d8ed63dbf47688fa929a3c290cf8d, State: Initialized, Role: FOLLOWER
I20260927 21:02:03.629813 20235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } }
I20260927 21:02:03.629902 20235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:03.629933 20235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:03.629981 20235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:03.630705 20235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } }
I20260927 21:02:03.630836 20235 leader_election.cc:302] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 412d8ed63dbf47688fa929a3c290cf8d; no voters:
I20260927 21:02:03.631034 20235 leader_election.cc:288] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:03.631062 20240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:03.631284 20240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [term 1 LEADER]: Becoming Leader. State: Replica: 412d8ed63dbf47688fa929a3c290cf8d, State: Running, Role: LEADER
I20260927 21:02:03.631419 20240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } }
I20260927 21:02:03.631740 20235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:03.631803 20241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "412d8ed63dbf47688fa929a3c290cf8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } } }
I20260927 21:02:03.631795 20240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 412d8ed63dbf47688fa929a3c290cf8d. Latest consensus state: current_term: 1 leader_uuid: "412d8ed63dbf47688fa929a3c290cf8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "412d8ed63dbf47688fa929a3c290cf8d" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36895 } } }
I20260927 21:02:03.631887 20241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:03.631894 20240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:03.632071 20244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:03.632581 20244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:03.634505 20244 catalog_manager.cc:1347] Generated new cluster ID: 26a489a73f0441d698ab59baeb275a39
I20260927 21:02:03.634554 20244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:03.638139 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:03.626330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.16.190.254:36895)
0927 21:02:03.626510 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:03.626520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:03.626555 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:02:03.627076 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:03.627083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:03.627133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:03.627251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:03.628100 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.629113 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.630293 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.630734 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.630868 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:03.634061 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:03.634066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:03.634067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:03.634080 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:03.636292 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.636785 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.636788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.636788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.636917 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.637169 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.637170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.637170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.637477 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:03.637658 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:03.637789 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:02:03.637924 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260927 21:02:03.642724 20244 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:03.643575 20244 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:03.657060 20244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 412d8ed63dbf47688fa929a3c290cf8d: Generated new TSK 0
I20260927 21:02:03.657310 20244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:03.685289 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:03.767470 20261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:03.767729 20261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:03.767769 20261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:03.769006 20261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:03.769078 20261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:03.769122 20261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:03.770586 20261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:03.770866 20261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:03.771113 20261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:03.776942 20267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:03.777029 20269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:03.777115 20261 server_base.cc:1034] running on GCE node
W20260927 21:02:03.777197 20266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:03.777421 20261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:03.777701 20261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:03.778879 20261 hybrid_clock.cc:648] HybridClock initialized: now 1790542923778876 us; error 39 us; skew 500 ppm
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:03.781495 20261 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:03.782014 20261 webserver.cc:460] Webserver started at http://127.16.190.193:45457/ using document root <none> and password file <none>
I20260927 21:02:03.782262 20261 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:03.782331 20261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:03.782483 20261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:03.783540 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "fcab0ffe575842dc8619fa34f0301a20"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "72559666523d9efbdb688c9d4d63a347"
server_key_iv: "5282fe6b2f37bfed06c9b687f00bc4be"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.783903 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "fcab0ffe575842dc8619fa34f0301a20"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "72559666523d9efbdb688c9d4d63a347"
server_key_iv: "5282fe6b2f37bfed06c9b687f00bc4be"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.785290 20261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:03.786051 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.786219 20261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:03.786322 20261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "fcab0ffe575842dc8619fa34f0301a20"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "72559666523d9efbdb688c9d4d63a347"
server_key_iv: "5282fe6b2f37bfed06c9b687f00bc4be"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.786396 20261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:03.809468 20261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:03.811939 20261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:03.812121 20261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:03.812384 20261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:03.812736 20261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:03.812794 20261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.812851 20261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:03.812881 20261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.819051 20261 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:45309
I20260927 21:02:03.819099 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:45309 every 8 connection(s)
I20260927 21:02:03.819432 20261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:03.820770 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20261
I20260927 21:02:03.820914 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:03.821152 17147 external_mini_cluster.cc:1428] Setting key 587fbc4c7817b4d1f142a6b76749896d
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:03.830626 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:03.820191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:60583 (local address 127.16.190.254:36895)
0927 21:02:03.820274 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:03.820276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:03.820477 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:02:03.821209 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:03.821210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:03.821224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:03.821278 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:03.822395 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.822937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.824528 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.824767 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.824918 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:03.827487 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:03.827488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:03.827489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:03.827497 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:03.829055 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.829564 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.829566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.829566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.829624 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.829888 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.829889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.829889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.830010 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:03.830097 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:03.830425 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 21:02:03.830523 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:02:03.830838 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:03.820150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36895 (local address 127.16.190.193:60583)
0927 21:02:03.820341 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:03.820387 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:03.821028 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:03.821366 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:03.821373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:03.822085 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:03.822295 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:03.822300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.823054 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.823056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:03.824429 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:03.824432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.824918 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.824919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:03.825080 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:03.825727 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:03.825739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:03.827370 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:03.829186 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:03.829193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:03.829196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:03.829492 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:03.829698 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:03.829699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:03.829699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:03.829815 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:03.830080 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:03.830087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:03.830273 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 21:02:03.830405 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:02:03.830592 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260927 21:02:03.831429 20390 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36895
I20260927 21:02:03.831712 20390 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:03.831936 20390 heartbeater.cc:511] Master 127.16.190.254:36895 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:02:03.832598 20199 ts_manager.cc:194] Registered new tserver with Master: fcab0ffe575842dc8619fa34f0301a20 (127.16.190.193:45309)
I20260927 21:02:03.833302 20199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:60583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:03.848448 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:03.931156 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:03.931387 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:03.931412 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:03.932955 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:03.933030 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:03.933081 20395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:03.934607 20395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:03.934821 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:03.935012 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:03.940270 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:03.940407 20395 server_base.cc:1034] running on GCE node
W20260927 21:02:03.940270 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:03.940346 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:03.940757 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:03.941090 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:03.942252 20395 hybrid_clock.cc:648] HybridClock initialized: now 1790542923942230 us; error 41 us; skew 500 ppm
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:03.944998 20395 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:03.945434 20395 webserver.cc:460] Webserver started at http://127.16.190.194:41747/ using document root <none> and password file <none>
I20260927 21:02:03.945632 20395 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:03.945674 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:03.945770 20395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:03.946808 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "841895d7ded746b5a0ed083188637ff3"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "3503e83e0b2f4ece332617bbaad4396d"
server_key_iv: "a10d49dae2c70b8b81b43475e9a4320e"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.947130 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "841895d7ded746b5a0ed083188637ff3"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "3503e83e0b2f4ece332617bbaad4396d"
server_key_iv: "a10d49dae2c70b8b81b43475e9a4320e"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.948446 20395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:02:03.949301 20410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.949460 20395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.949579 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "841895d7ded746b5a0ed083188637ff3"
format_stamp: "Formatted at 2026-09-27 21:02:03 on dist-test-slave-6c7s"
server_key: "3503e83e0b2f4ece332617bbaad4396d"
server_key_iv: "a10d49dae2c70b8b81b43475e9a4320e"
server_key_version: "encryptionkey@0"
I20260927 21:02:03.949641 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:03.962546 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:03.964654 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:03.964772 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:03.965041 20395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:03.965317 20395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:03.965349 20395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.965371 20395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:03.965384 20395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:03.970974 20395 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:37211
I20260927 21:02:03.971057 20523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:37211 every 8 connection(s)
I20260927 21:02:03.971392 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:03.973737 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20395
I20260927 21:02:03.973850 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:03.974037 17147 external_mini_cluster.cc:1428] Setting key 1f29c214210564e4190c3d9180fe1347
Sep 27 21:02:03 dist-test-slave-6c7s krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542923, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
I20260927 21:02:03.982723 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:03.972239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:45597 (local address 127.16.190.254:36895)
0927 21:02:03.972322 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:03.972324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:03.972536 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:02:03.973077 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:03.973078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:03.973088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:03.973162 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:03.974570 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.975345 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.976742 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.976998 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.977066 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:03.979676 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:03.979678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:03.979678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:03.979687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:03.981254 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.981711 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.981712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.981712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.981775 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:03.982037 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:03.982038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:03.982039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:03.982186 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:03.982248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:03.982537 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:02:03.982620 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:02:03.982954 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:03.972161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36895 (local address 127.16.190.194:45597)
0927 21:02:03.972393 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:03.972435 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:03.972957 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:03.973249 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:03.973252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:03.974092 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:03.974415 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:03.974422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.975449 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.975450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:03.976647 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:03.976650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:03.977087 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:03.977088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:03.977238 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:03.977852 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:03.977863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:03.979565 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:03.981358 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:03.981366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:03.981368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:03.981614 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:03.981850 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:03.981850 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:03.981851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:03.981968 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:03.982248 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:03.982255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:03.982402 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:02:03.982532 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 21:02:03.982705 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260927 21:02:03.983491 20524 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36895
I20260927 21:02:03.983593 20524 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:03.983747 20524 heartbeater.cc:511] Master 127.16.190.254:36895 requested a full tablet report, sending...
I20260927 21:02:03.984153 20199 ts_manager.cc:194] Registered new tserver with Master: 841895d7ded746b5a0ed083188637ff3 (127.16.190.194:37211)
I20260927 21:02:03.984587 20199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:45597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:04.000375 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:04.088099 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:04.088315 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:04.088338 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:04.089686 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:04.089733 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:04.089764 20529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:04.091274 20529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:04.091470 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:04.091650 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:04.097221 20535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:04.097236 20529 server_base.cc:1034] running on GCE node
W20260927 21:02:04.097224 20534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:04.097224 20537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:04.097746 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:04.098009 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:04.099175 20529 hybrid_clock.cc:648] HybridClock initialized: now 1790542924099161 us; error 29 us; skew 500 ppm
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542924, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:04.101876 20529 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:04.102339 20529 webserver.cc:460] Webserver started at http://127.16.190.195:42331/ using document root <none> and password file <none>
I20260927 21:02:04.102574 20529 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:04.102620 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:04.102779 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:04.103977 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "e7344a22247346dfb58439ee4d9c4c80"
format_stamp: "Formatted at 2026-09-27 21:02:04 on dist-test-slave-6c7s"
server_key: "41c3d9e7daeb86c63eb6c6e5593c9587"
server_key_iv: "d7f45a94055ceb46d2ac53a1a3246138"
server_key_version: "encryptionkey@0"
I20260927 21:02:04.104367 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "e7344a22247346dfb58439ee4d9c4c80"
format_stamp: "Formatted at 2026-09-27 21:02:04 on dist-test-slave-6c7s"
server_key: "41c3d9e7daeb86c63eb6c6e5593c9587"
server_key_iv: "d7f45a94055ceb46d2ac53a1a3246138"
server_key_version: "encryptionkey@0"
I20260927 21:02:04.105844 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:04.106608 20544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:04.106803 20529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:04.106886 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "e7344a22247346dfb58439ee4d9c4c80"
format_stamp: "Formatted at 2026-09-27 21:02:04 on dist-test-slave-6c7s"
server_key: "41c3d9e7daeb86c63eb6c6e5593c9587"
server_key_iv: "d7f45a94055ceb46d2ac53a1a3246138"
server_key_version: "encryptionkey@0"
I20260927 21:02:04.106941 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:04.121351 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:04.123317 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:04.123428 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:04.123615 20529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:04.123924 20529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:04.123956 20529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:04.123978 20529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:04.124024 20529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:04.129835 20529 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:36539
I20260927 21:02:04.129912 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:36539 every 8 connection(s)
I20260927 21:02:04.130221 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542924, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:04.138360 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20529
I20260927 21:02:04.138470 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:04.138727 17147 external_mini_cluster.cc:1428] Setting key 6be9f3cdf0c1acec149ceccf7316bfad
I20260927 21:02:04.140766 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:04.130995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:49721 (local address 127.16.190.254:36895)
0927 21:02:04.131065 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:04.131067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:04.131298 (+ 231us) server_negotiation.cc:407] Connection header received
0927 21:02:04.131781 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:04.131782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:04.131792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:04.131847 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:04.133028 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.133628 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.134702 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.134894 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.134962 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:04.137520 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:04.137522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:04.137522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:04.137530 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:04.139383 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:04.139916 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:04.139917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:04.139917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:04.139971 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:04.140198 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:04.140199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:04.140199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:04.140306 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:04.140358 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:04.140635 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 21:02:04.140701 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:04.141047 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:04.130979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36895 (local address 127.16.190.195:49721)
0927 21:02:04.131173 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:04.131210 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:04.131697 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:04.131916 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:04.131919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:04.132662 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:04.132939 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:04.132944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.133746 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.133748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:04.134602 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:04.134605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.134986 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.134987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:04.135117 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:04.135758 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:04.135768 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:04.137396 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:04.139464 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:04.139472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:04.139476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:04.139825 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:04.140037 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:04.140037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:04.140038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:04.140140 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:04.140362 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:04.140365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:04.140482 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 21:02:04.140609 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 21:02:04.140733 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260927 21:02:04.141482 20658 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36895
I20260927 21:02:04.141579 20658 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:04.141719 20658 heartbeater.cc:511] Master 127.16.190.254:36895 requested a full tablet report, sending...
I20260927 21:02:04.142071 20199 ts_manager.cc:194] Registered new tserver with Master: e7344a22247346dfb58439ee4d9c4c80 (127.16.190.195:36539)
I20260927 21:02:04.142403 20199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:49721
I20260927 21:02:04.150826 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542924, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542924, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:04.173643 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:04.166475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.16.190.254:36895)
0927 21:02:04.166548 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:04.166551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:04.166677 (+ 126us) server_negotiation.cc:407] Connection header received
0927 21:02:04.166726 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:04.166727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:04.166735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:04.166796 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:04.167368 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.167873 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.168612 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:04.168801 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:04.168949 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:04.171194 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:04.171196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:04.171196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:04.171203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:04.172483 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:04.172894 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:04.172895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:04.172895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:04.172949 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:04.173121 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:04.173122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:04.173122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:04.173270 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:04.173347 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:04.173488 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 21:02:04.173559 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 21:02:04.173905 20199 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46466
I20260927 21:02:04.174464 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20261
I20260927 21:02:04.179992 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20395
I20260927 21:02:04.184302 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20529
I20260927 21:02:04.188608 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20167
2026-09-27T21:02:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1879 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20670](info): set up 2 sockets
Sep 27 21:02:04 dist-test-slave-6c7s krb5kdc[20670](info): commencing operation
krb5kdc: starting...
W20260927 21:02:05.245746 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:05.368587 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39607
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:05.449812 20683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:05.450075 20683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:05.450119 20683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:05.451284 20683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:05.451359 20683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:05.451381 20683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:05.451400 20683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:05.451432 20683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:05.452984 20683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:05.453274 20683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:05.453512 20683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:05.458791 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.458804 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.458792 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:05.458971 20683 server_base.cc:1034] running on GCE node
I20260927 21:02:05.459275 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:05.459630 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:05.460844 20683 hybrid_clock.cc:648] HybridClock initialized: now 1790542925460777 us; error 48 us; skew 500 ppm
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:05.463266 20683 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:05.463757 20683 webserver.cc:460] Webserver started at http://127.16.190.254:39623/ using document root <none> and password file <none>
I20260927 21:02:05.463989 20683 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:05.464056 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:05.464200 20683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:05.465354 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "3b9ed89d06ad43aab79f8c82ecc010d2"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "30e285677985b674ff081f525c2cba9c"
server_key_iv: "b2e4baa8ba43b3dd8c8395cbd48c7511"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.465734 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "3b9ed89d06ad43aab79f8c82ecc010d2"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "30e285677985b674ff081f525c2cba9c"
server_key_iv: "b2e4baa8ba43b3dd8c8395cbd48c7511"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.467089 20683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:05.467862 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.468065 20683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:05.468179 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "3b9ed89d06ad43aab79f8c82ecc010d2"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "30e285677985b674ff081f525c2cba9c"
server_key_iv: "b2e4baa8ba43b3dd8c8395cbd48c7511"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.468257 20683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:05.481914 20683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:05.484143 20683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:05.484338 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:05.488535 20683 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39607
I20260927 21:02:05.488569 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39607 every 8 connection(s)
I20260927 21:02:05.489140 20683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:05.489816 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:05.492146 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2: Bootstrap starting.
I20260927 21:02:05.492690 20751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:05.493011 20751 log.cc:824] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:05.493662 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20683
I20260927 21:02:05.493762 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2: No bootstrap required, opened a new log
I20260927 21:02:05.493769 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:05.493968 17147 external_mini_cluster.cc:1428] Setting key 1ac8af4d53af9c5ed5223578760690b6
I20260927 21:02:05.496184 20751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } }
I20260927 21:02:05.496326 20751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:05.496348 20751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9ed89d06ad43aab79f8c82ecc010d2, State: Initialized, Role: FOLLOWER
I20260927 21:02:05.496549 20751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } }
I20260927 21:02:05.496641 20751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:05.496677 20751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:05.496733 20751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:05.497510 20751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } }
I20260927 21:02:05.497604 20751 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b9ed89d06ad43aab79f8c82ecc010d2; no voters:
I20260927 21:02:05.497818 20751 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:05.497850 20756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:05.497952 20756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9ed89d06ad43aab79f8c82ecc010d2, State: Running, Role: LEADER
I20260927 21:02:05.498034 20756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } }
I20260927 21:02:05.498082 20751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:05.498364 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } } }
I20260927 21:02:05.498522 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:05.498723 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9ed89d06ad43aab79f8c82ecc010d2. Latest consensus state: current_term: 1 leader_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9ed89d06ad43aab79f8c82ecc010d2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39607 } } }
I20260927 21:02:05.498793 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:05.499145 20764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:05.499681 20764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:05.501319 20764 catalog_manager.cc:1347] Generated new cluster ID: 711887752c784caa8be4b55d9b8b2865
I20260927 21:02:05.501375 20764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:05.507047 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.494568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.16.190.254:39607)
0927 21:02:05.494753 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:05.494761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:05.494787 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:02:05.496175 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:05.496185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:05.496247 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:05.496428 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:05.497274 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.498634 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.499716 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.500314 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.500430 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.503407 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:05.503411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:05.503411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:05.503422 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:05.505380 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.505832 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.505834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.505835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.505948 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.506163 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.506164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.506165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.506458 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:05.506608 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:05.506725 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:02:05.506851 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260927 21:02:05.512331 20764 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:05.513105 20764 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:05.518481 20764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b9ed89d06ad43aab79f8c82ecc010d2: Generated new TSK 0
I20260927 21:02:05.518689 20764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:05.547492 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:05.631997 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:05.632213 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:05.632237 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:05.633491 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:05.633538 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:05.633574 20777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:05.635110 20777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:05.635365 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:05.635632 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:05.641369 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:05.641489 20777 server_base.cc:1034] running on GCE node
W20260927 21:02:05.641427 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.641390 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:05.641814 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:05.642091 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:05.643258 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790542925643239 us; error 37 us; skew 500 ppm
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:05.645821 20777 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:05.646301 20777 webserver.cc:460] Webserver started at http://127.16.190.193:37085/ using document root <none> and password file <none>
I20260927 21:02:05.646535 20777 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:05.646611 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:05.646770 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:05.647861 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "70106ba02c1a42b7b9ce9c83071d1776"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "343371b8487dbf6034c78aa2d54462fa"
server_key_iv: "401351100f62d9746a72ba5764f955a1"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.648242 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "70106ba02c1a42b7b9ce9c83071d1776"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "343371b8487dbf6034c78aa2d54462fa"
server_key_iv: "401351100f62d9746a72ba5764f955a1"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.649752 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:05.650652 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.650941 20777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:05.651054 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "70106ba02c1a42b7b9ce9c83071d1776"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "343371b8487dbf6034c78aa2d54462fa"
server_key_iv: "401351100f62d9746a72ba5764f955a1"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.651144 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:05.658537 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:05.660668 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:05.660892 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:05.661142 20777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:05.661450 20777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:05.661517 20777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.661583 20777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:05.661615 20777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.667557 20777 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:36085
I20260927 21:02:05.667661 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:36085 every 8 connection(s)
I20260927 21:02:05.668049 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:05.672674 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20777
I20260927 21:02:05.672791 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:05.673017 17147 external_mini_cluster.cc:1428] Setting key 1e195b926257954a1eeda088ff6e48d0
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:05.678998 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.668868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:33115 (local address 127.16.190.254:39607)
0927 21:02:05.668963 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:05.668967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:05.669148 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:02:05.670146 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:05.670147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:05.670168 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:05.670221 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:05.670940 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.671495 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.672663 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.673096 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.673194 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.675951 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:05.675953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:05.675953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:05.675960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:05.677513 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.677950 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.677951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.677952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.678017 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.678286 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.678287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.678287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.678414 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:05.678488 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:05.678800 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 21:02:05.678904 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:05.679219 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.668822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39607 (local address 127.16.190.193:33115)
0927 21:02:05.669020 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:05.669065 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:05.670022 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:05.670296 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:05.670300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:05.670567 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:05.670832 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.670844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.671599 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.671600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.672553 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.672556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.673201 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.673203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.673455 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.674143 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:05.674152 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:05.675833 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:05.677617 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.677626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.677628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.677873 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.678089 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.678090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.678090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.678210 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.678489 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:05.678496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:05.678643 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:02:05.678753 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:02:05.678928 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260927 21:02:05.679888 20906 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39607
I20260927 21:02:05.680002 20906 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:05.680213 20906 heartbeater.cc:511] Master 127.16.190.254:39607 requested a full tablet report, sending...
I20260927 21:02:05.680835 20715 ts_manager.cc:194] Registered new tserver with Master: 70106ba02c1a42b7b9ce9c83071d1776 (127.16.190.193:36085)
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:05.681823 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:33115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:05.700202 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:05.780475 20911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:05.780684 20911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:05.780709 20911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:05.781899 20911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:05.781946 20911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:05.781973 20911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:05.783458 20911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:05.783701 20911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:05.783936 20911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:05.789264 20919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.789285 20917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.789264 20916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:05.789363 20911 server_base.cc:1034] running on GCE node
I20260927 21:02:05.789733 20911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:05.789969 20911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:05.791118 20911 hybrid_clock.cc:648] HybridClock initialized: now 1790542925791101 us; error 30 us; skew 500 ppm
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:05.793727 20911 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:05.794293 20911 webserver.cc:460] Webserver started at http://127.16.190.194:43303/ using document root <none> and password file <none>
I20260927 21:02:05.794498 20911 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:05.794540 20911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:05.794637 20911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:05.795746 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "1ce4b80545d04eae8549b9d944f6c762"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "ca820f383702c9b0db30c62d42ce2cf3"
server_key_iv: "cfceb1d837115249d738f49386bb2eaf"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.796070 20911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "1ce4b80545d04eae8549b9d944f6c762"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "ca820f383702c9b0db30c62d42ce2cf3"
server_key_iv: "cfceb1d837115249d738f49386bb2eaf"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.797463 20911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:02:05.798257 20926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.798432 20911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:05.798535 20911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "1ce4b80545d04eae8549b9d944f6c762"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "ca820f383702c9b0db30c62d42ce2cf3"
server_key_iv: "cfceb1d837115249d738f49386bb2eaf"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.798626 20911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:05.806052 20911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:05.808280 20911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:05.808449 20911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:05.808746 20911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:05.809125 20911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:05.809188 20911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.809233 20911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:05.809276 20911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.815176 20911 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:35761
I20260927 21:02:05.815223 21039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:35761 every 8 connection(s)
I20260927 21:02:05.815555 20911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:05.825047 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 20911
I20260927 21:02:05.825177 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:05.825256 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.816395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:49657 (local address 127.16.190.254:39607)
0927 21:02:05.816473 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:05.816475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:05.816706 (+ 231us) server_negotiation.cc:407] Connection header received
0927 21:02:05.817771 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:05.817772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:05.817783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:05.817839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:05.818475 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.818973 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.820027 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.820207 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.820258 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.822562 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:05.822564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:05.822564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:05.822570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:05.823842 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.824223 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.824224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.824224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.824279 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.824505 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.824506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.824506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.824647 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:05.824702 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:05.825068 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:02:05.825162 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:02:05.825403 17147 external_mini_cluster.cc:1428] Setting key e0a825121d28e39af11aec0768e406d9
I20260927 21:02:05.825456 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.816370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39607 (local address 127.16.190.194:49657)
0927 21:02:05.816568 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:05.816616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:05.817669 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:05.817907 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:05.817911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:05.818197 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:05.818401 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.818405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.819084 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.819085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.819943 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.819946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.820286 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.820287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.820391 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.821011 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:05.821020 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:05.822450 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:05.823923 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.823929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.823931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.824151 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.824345 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.824346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.824346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.824447 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.824710 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:05.824715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:05.824905 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 21:02:05.825033 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:02:05.825192 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260927 21:02:05.826108 21040 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39607
I20260927 21:02:05.826223 21040 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:05.826443 21040 heartbeater.cc:511] Master 127.16.190.254:39607 requested a full tablet report, sending...
I20260927 21:02:05.826843 20715 ts_manager.cc:194] Registered new tserver with Master: 1ce4b80545d04eae8549b9d944f6c762 (127.16.190.194:35761)
I20260927 21:02:05.827364 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:49657
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:05.853368 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:05.936029 21045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:05.936288 21045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:05.936314 21045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:05.938028 21045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:05.938092 21045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:05.938122 21045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:05.939741 21045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:05.940021 21045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:05.940306 21045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:05.945905 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.945902 21051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:05.945969 21053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:05.946442 21045 server_base.cc:1034] running on GCE node
I20260927 21:02:05.946713 21045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:05.947018 21045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:05.948179 21045 hybrid_clock.cc:648] HybridClock initialized: now 1790542925948166 us; error 29 us; skew 500 ppm
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:05.951027 21045 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:05.951546 21045 webserver.cc:460] Webserver started at http://127.16.190.195:44513/ using document root <none> and password file <none>
I20260927 21:02:05.951828 21045 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:05.951906 21045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:05.952068 21045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:05.953339 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "f3023916e4844c52a8e97cfc0628450a"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "eab4c4072ed2a5da7b048f39af1f7453"
server_key_iv: "e89557f7893689c166d6725324e3dddd"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.953745 21045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "f3023916e4844c52a8e97cfc0628450a"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "eab4c4072ed2a5da7b048f39af1f7453"
server_key_iv: "e89557f7893689c166d6725324e3dddd"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.955323 21045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:05.956103 21060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.956329 21045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:05.956451 21045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "f3023916e4844c52a8e97cfc0628450a"
format_stamp: "Formatted at 2026-09-27 21:02:05 on dist-test-slave-6c7s"
server_key: "eab4c4072ed2a5da7b048f39af1f7453"
server_key_iv: "e89557f7893689c166d6725324e3dddd"
server_key_version: "encryptionkey@0"
I20260927 21:02:05.956557 21045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:05.965595 21045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:05.967808 21045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:05.967962 21045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:05.968199 21045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:05.968550 21045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:05.968585 21045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.968608 21045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:05.968621 21045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:05.974946 21045 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:46637
I20260927 21:02:05.975011 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:46637 every 8 connection(s)
I20260927 21:02:05.975320 21045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:05.978693 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21045
I20260927 21:02:05.978796 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:05.978982 17147 external_mini_cluster.cc:1428] Setting key c09eee2d04f88ff0512ea51385355e79
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:05.985287 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.976102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:35503 (local address 127.16.190.254:39607)
0927 21:02:05.976191 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:05.976194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:05.976372 (+ 178us) server_negotiation.cc:407] Connection header received
0927 21:02:05.977356 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:05.977357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:05.977369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:05.977435 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:05.978197 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.978917 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.980283 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.980468 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.980528 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.982859 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:05.982860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:05.982861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:05.982867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:05.984063 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.984429 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.984430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.984430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.984478 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:05.984703 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:05.984703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:05.984704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:05.984805 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:05.984916 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:05.985156 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:02:05.985219 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 21:02:05.985600 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:05.976063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39607 (local address 127.16.190.195:35503)
0927 21:02:05.976250 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:05.976287 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:05.977209 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:05.977600 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:05.977604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:05.977901 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:05.978099 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.978104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.979008 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.979010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.980199 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:05.980202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:05.980548 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:05.980549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:05.980689 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:05.981329 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:05.981337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:05.982761 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:05.984138 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.984144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.984146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.984365 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.984533 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:05.984534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:05.984534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:05.984627 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:05.984908 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:05.984912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:05.985029 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 21:02:05.985115 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:02:05.985236 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260927 21:02:05.986207 21174 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39607
I20260927 21:02:05.986321 21174 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:05.986501 21174 heartbeater.cc:511] Master 127.16.190.254:39607 requested a full tablet report, sending...
I20260927 21:02:05.986971 20715 ts_manager.cc:194] Registered new tserver with Master: f3023916e4844c52a8e97cfc0628450a (127.16.190.195:46637)
I20260927 21:02:05.987399 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:35503
I20260927 21:02:05.990814 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 21:02:05 dist-test-slave-6c7s krb5kdc[20670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:02:06 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:02:06.015978 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:06.007602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.16.190.193:36085)
0927 21:02:06.007769 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:06.007775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:06.007794 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:02:06.007821 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:06.007823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:06.007856 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:06.007949 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:06.008514 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:06.009349 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:06.010212 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:06.010528 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:06.010589 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:06.013204 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:06.013211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:06.013212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:06.013219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:06.014846 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:06.015208 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:06.015212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:06.015212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:06.015285 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:06.015521 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:06.015521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:06.015522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:06.015663 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:06.015788 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:06.015832 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 21:02:06.015897 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
Sep 27 21:02:06 dist-test-slave-6c7s krb5kdc[20670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:06.025599 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:06.017365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.16.190.254:39607)
0927 21:02:06.017414 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:06.017416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:06.017460 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:02:06.017600 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:06.017601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:06.017610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:06.017659 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:06.018159 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:06.018757 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:06.019484 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:06.019687 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:06.019746 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:06.022244 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:06.022246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:06.022246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:06.022256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:06.024068 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:06.024512 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:06.024513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:06.024514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:06.024588 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:06.024847 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:06.024848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:06.024849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:06.025088 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:06.025206 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:06.025377 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:02:06.025472 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 21:02:06.025902 20715 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35986
I20260927 21:02:06.026307 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20777
I20260927 21:02:06.031668 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20911
I20260927 21:02:06.036512 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21045
I20260927 21:02:06.041080 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 20683
2026-09-27T21:02:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1852 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:06 dist-test-slave-6c7s krb5kdc[21186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:06 dist-test-slave-6c7s krb5kdc[21186](info): set up 2 sockets
Sep 27 21:02:06 dist-test-slave-6c7s krb5kdc[21186](info): commencing operation
krb5kdc: starting...
W20260927 21:02:07.109694 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:07.228266 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45379
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:07.310652 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:07.310874 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:07.310899 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:07.312073 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:07.312116 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:07.312131 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:07.312145 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:07.312157 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:07.313766 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:07.314033 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:07.314270 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:07.319228 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:07.319228 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.319339 21199 server_base.cc:1034] running on GCE node
W20260927 21:02:07.319475 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.319731 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:07.320024 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:07.321161 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790542927321147 us; error 29 us; skew 500 ppm
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:07.329985 21199 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:07.330324 21199 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:07.330370 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:07.330464 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:07.331555 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "663482b25c3f402aad8820f8712d4d2c"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "1a57c1510620f47904f8f77945e786f8"
server_key_iv: "014c20aea311a9f7bd34e0de19b7762e"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.331881 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "663482b25c3f402aad8820f8712d4d2c"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "1a57c1510620f47904f8f77945e786f8"
server_key_iv: "014c20aea311a9f7bd34e0de19b7762e"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.333246 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:02:07.333962 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.334180 21199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:07.334273 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "663482b25c3f402aad8820f8712d4d2c"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "1a57c1510620f47904f8f77945e786f8"
server_key_iv: "014c20aea311a9f7bd34e0de19b7762e"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.334342 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:07.347800 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:07.350191 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:07.350315 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:07.354471 21199 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:45379
I20260927 21:02:07.354501 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:45379 every 8 connection(s)
I20260927 21:02:07.354944 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:07.355945 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:07.358394 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: Bootstrap starting.
I20260927 21:02:07.359014 21266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:07.359341 21266 log.cc:824] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:07.360060 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: No bootstrap required, opened a new log
I20260927 21:02:07.361843 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } }
I20260927 21:02:07.361990 21266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:07.362013 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663482b25c3f402aad8820f8712d4d2c, State: Initialized, Role: FOLLOWER
I20260927 21:02:07.362174 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } }
I20260927 21:02:07.362289 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:07.362315 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:07.362363 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:07.363284 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } }
I20260927 21:02:07.363344 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21199
I20260927 21:02:07.363409 21266 leader_election.cc:302] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 663482b25c3f402aad8820f8712d4d2c; no voters:
I20260927 21:02:07.363441 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:07.363592 21266 leader_election.cc:288] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:07.363642 17147 external_mini_cluster.cc:1428] Setting key 307deb7b2c0ade532ed2dd536fcdacd2
I20260927 21:02:07.363672 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:07.363850 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [term 1 LEADER]: Becoming Leader. State: Replica: 663482b25c3f402aad8820f8712d4d2c, State: Running, Role: LEADER
I20260927 21:02:07.363862 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:07.363981 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } }
I20260927 21:02:07.364535 21271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 663482b25c3f402aad8820f8712d4d2c. Latest consensus state: current_term: 1 leader_uuid: "663482b25c3f402aad8820f8712d4d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } } }
I20260927 21:02:07.364682 21271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:07.364871 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "663482b25c3f402aad8820f8712d4d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "663482b25c3f402aad8820f8712d4d2c" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45379 } } }
I20260927 21:02:07.364938 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:07.365094 21279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:07.365581 21279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:07.367241 21279 catalog_manager.cc:1347] Generated new cluster ID: f94f5a1017204d59a36ea360a8284d89
I20260927 21:02:07.367298 21279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:07.376368 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.364406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.16.190.254:45379)
0927 21:02:07.364738 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.364748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.364780 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:02:07.364929 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.364937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.365002 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.365752 (+ 750us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:07.366009 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.367269 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.368379 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.369024 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.369133 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.371903 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:07.371907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:07.371907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:07.371921 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:07.374627 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.375031 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.375035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.375035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.375151 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.375357 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.375358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.375358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.375698 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:07.375862 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.375976 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.376129 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260927 21:02:07.380385 21279 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:07.381129 21279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:07.387567 21279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: Generated new TSK 0
I20260927 21:02:07.387805 21279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:07.428938 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:07.523110 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:07.523330 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:07.523356 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:07.524653 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:07.524700 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:07.524730 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:07.526314 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:07.526573 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:07.526825 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:07.532202 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.532258 21292 server_base.cc:1034] running on GCE node
W20260927 21:02:07.532228 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:07.532222 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.532704 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:07.533001 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:07.534162 21292 hybrid_clock.cc:648] HybridClock initialized: now 1790542927534142 us; error 49 us; skew 500 ppm
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:07.536873 21292 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:07.537258 21292 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:07.537307 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:07.537401 21292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:07.538506 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "72ca2824d2884632bb4120cb05953a15"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "317f26d4aa640fd9dc8ad4d286bbfd69"
server_key_iv: "1baf7b00ecf478251e803cd58ddf5b73"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.538883 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "72ca2824d2884632bb4120cb05953a15"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "317f26d4aa640fd9dc8ad4d286bbfd69"
server_key_iv: "1baf7b00ecf478251e803cd58ddf5b73"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.540252 21292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:07.541136 21306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.541359 21292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:07.541451 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "72ca2824d2884632bb4120cb05953a15"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "317f26d4aa640fd9dc8ad4d286bbfd69"
server_key_iv: "1baf7b00ecf478251e803cd58ddf5b73"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.541512 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:07.562004 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:07.564376 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:07.564522 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:07.564855 21292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:07.565241 21292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:07.565276 21292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.565299 21292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:07.565311 21292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.571321 21292 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:38519
I20260927 21:02:07.571385 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:38519 every 8 connection(s)
I20260927 21:02:07.571697 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:07.575215 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21292
I20260927 21:02:07.575378 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:07.575562 17147 external_mini_cluster.cc:1428] Setting key 1b550cfe804e25f3f6a0fef8ac91d743
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:07.582885 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.572456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:35523 (local address 127.16.190.254:45379)
0927 21:02:07.572561 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.572563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.572788 (+ 225us) server_negotiation.cc:407] Connection header received
0927 21:02:07.573382 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.573383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.573395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.573449 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:07.574450 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.574967 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.576294 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.576580 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.576662 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.579613 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:07.579614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:07.579615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:07.579625 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:07.581327 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.581808 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.581810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.581810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.581877 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.582169 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.582170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.582170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.582300 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:07.582380 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.582690 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.582783 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:02:07.583122 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.572431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45379 (local address 127.16.190.193:35523)
0927 21:02:07.572649 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.572693 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.573270 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.573535 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.573540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.573825 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:07.574335 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.574341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.575083 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.575084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.576224 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.576226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.576688 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.576689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.576879 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.577504 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:07.577513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:07.579483 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:07.581444 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.581453 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.581456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.581718 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.581958 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.581959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.581960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.582091 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.582372 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:07.582379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:07.582547 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.582667 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.582851 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260927 21:02:07.583833 21420 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45379
I20260927 21:02:07.583966 21420 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:07.584167 21420 heartbeater.cc:511] Master 127.16.190.254:45379 requested a full tablet report, sending...
I20260927 21:02:07.584908 21230 ts_manager.cc:194] Registered new tserver with Master: 72ca2824d2884632bb4120cb05953a15 (127.16.190.193:38519)
I20260927 21:02:07.585677 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:35523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:07.604609 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:07.687877 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:07.688124 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:07.688153 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:07.689486 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:07.689530 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:07.689559 21425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:07.691114 21425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:07.691339 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:07.691533 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:07.696775 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.696866 21425 server_base.cc:1034] running on GCE node
W20260927 21:02:07.696943 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:07.696801 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.697260 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:07.697536 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:07.698750 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790542927698699 us; error 61 us; skew 500 ppm
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:07.701727 21425 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:07.702126 21425 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:07.702178 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:07.702273 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:07.703316 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "fa1dc5ca38a74eaca254b55616a258fb"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "eb8227a5518889d4fb376c4db6a62704"
server_key_iv: "dc536ed56cb884293d0b6897d182be10"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.703634 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "fa1dc5ca38a74eaca254b55616a258fb"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "eb8227a5518889d4fb376c4db6a62704"
server_key_iv: "dc536ed56cb884293d0b6897d182be10"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.705075 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:07.705960 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.706193 21425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:07.706279 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "fa1dc5ca38a74eaca254b55616a258fb"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "eb8227a5518889d4fb376c4db6a62704"
server_key_iv: "dc536ed56cb884293d0b6897d182be10"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.706337 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:07.716027 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:07.718281 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:07.718398 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:07.718592 21425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:07.718914 21425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:07.718950 21425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.718972 21425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:07.718986 21425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.724658 21425 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:33437
I20260927 21:02:07.724864 21552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:33437 every 8 connection(s)
I20260927 21:02:07.725078 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:07.729808 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21425
I20260927 21:02:07.729892 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:07.730093 17147 external_mini_cluster.cc:1428] Setting key c1a80d8f7ba2a3fed11d46679c8c0d2e
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:07.735219 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.725903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:60123 (local address 127.16.190.254:45379)
0927 21:02:07.726008 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.726010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.726181 (+ 171us) server_negotiation.cc:407] Connection header received
0927 21:02:07.726761 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.726762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.726775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.726828 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:07.727825 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.728328 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.729409 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.729611 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.729719 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.732010 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:07.732011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:07.732012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:07.732027 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:07.733692 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.734235 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.734237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.734237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.734305 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.734570 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.734571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.734571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.734722 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:07.734797 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.735051 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.735144 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:02:07.735503 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.725856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45379 (local address 127.16.190.194:60123)
0927 21:02:07.726054 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.726098 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.726650 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.726917 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.726921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.727204 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:07.727715 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.727721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.728421 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.728422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.729320 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.729323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.729725 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.729726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.729848 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.730483 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:07.730493 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:07.731912 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:07.733799 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.733806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.733808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.734163 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.734385 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.734386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.734386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.734503 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.734791 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:07.734795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:07.734914 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.734998 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.735220 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260927 21:02:07.736033 21553 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45379
I20260927 21:02:07.736153 21553 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:07.736305 21553 heartbeater.cc:511] Master 127.16.190.254:45379 requested a full tablet report, sending...
I20260927 21:02:07.736713 21230 ts_manager.cc:194] Registered new tserver with Master: fa1dc5ca38a74eaca254b55616a258fb (127.16.190.194:33437)
I20260927 21:02:07.737128 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:60123
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:07.759524 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:43831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:07.843323 21558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:07.843582 21558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:07.843608 21558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:07.844915 21558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:07.844959 21558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:07.844987 21558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:07.846449 21558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:43831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:07.846643 21558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:07.846828 21558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:07.851977 21564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:07.851997 21563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:07.852003 21566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:07.852263 21558 server_base.cc:1034] running on GCE node
I20260927 21:02:07.852572 21558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:07.852880 21558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:07.854102 21558 hybrid_clock.cc:648] HybridClock initialized: now 1790542927854085 us; error 51 us; skew 500 ppm
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:07.856906 21558 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:07.857337 21558 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:07.857422 21558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:07.857589 21558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:07.858747 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "9b84e4b906724760884f2ddeaa695992"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "ce3ecff00beb0d4b5e8fafe2f98e7202"
server_key_iv: "7d69d07a0855340ea73467956384780a"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.859146 21558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "9b84e4b906724760884f2ddeaa695992"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "ce3ecff00beb0d4b5e8fafe2f98e7202"
server_key_iv: "7d69d07a0855340ea73467956384780a"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.860561 21558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260927 21:02:07.861446 21572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.861634 21558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:07.861735 21558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "9b84e4b906724760884f2ddeaa695992"
format_stamp: "Formatted at 2026-09-27 21:02:07 on dist-test-slave-6c7s"
server_key: "ce3ecff00beb0d4b5e8fafe2f98e7202"
server_key_iv: "7d69d07a0855340ea73467956384780a"
server_key_version: "encryptionkey@0"
I20260927 21:02:07.861825 21558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:07.878567 21558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:07.881202 21558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:07.881386 21558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:07.881821 21558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:07.882186 21558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:07.882246 21558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.882292 21558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:07.882318 21558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:07.888443 21558 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34143
I20260927 21:02:07.888477 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34143 every 8 connection(s)
I20260927 21:02:07.888782 21558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:02:07 dist-test-slave-6c7s krb5kdc[21186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542927, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:07.895444 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21558
I20260927 21:02:07.895512 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:07.895718 17147 external_mini_cluster.cc:1428] Setting key e414e5da21c1276174a585c8d3a45828
I20260927 21:02:07.898535 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.889621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:44851 (local address 127.16.190.254:45379)
0927 21:02:07.889711 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.889714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.889872 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:02:07.890377 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.890378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.890388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.890450 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:07.891418 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.891952 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.893044 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.893235 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.893326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.895738 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:07.895740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:07.895740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:07.895748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:07.897314 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.897745 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.897745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.897746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.897800 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.898011 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.898011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.898011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.898121 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:07.898172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.898407 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.898469 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:02:07.898900 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.889580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45379 (local address 127.16.190.195:44851)
0927 21:02:07.889747 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.889783 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.890269 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.890517 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.890521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.890794 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:07.891323 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.891329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.892034 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.892034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.892945 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.892948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.893395 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.893396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.893513 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.894149 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:07.894158 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:07.895647 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:07.897386 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.897392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.897394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.897675 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.897859 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:07.897860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:07.897860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:07.897957 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:07.898176 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:07.898180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:07.898294 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.898378 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.898499 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260927 21:02:07.899453 21686 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45379
I20260927 21:02:07.899562 21686 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:07.899770 21686 heartbeater.cc:511] Master 127.16.190.254:45379 requested a full tablet report, sending...
I20260927 21:02:07.900167 21230 ts_manager.cc:194] Registered new tserver with Master: 9b84e4b906724760884f2ddeaa695992 (127.16.190.195:34143)
I20260927 21:02:07.900537 21230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:44851
I20260927 21:02:07.907505 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:02:07.915385 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.908877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.16.190.254:45379)
0927 21:02:07.908933 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.908936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.909073 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:02:07.909197 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.909198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.909210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.909255 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:07.909852 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.910468 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.911208 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.911415 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.911499 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.912740 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:07.912742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:07.912742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:07.912749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:07.914145 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.914543 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.914544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.914545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.914604 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:07.914822 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:07.914823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:07.914823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:07.914976 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:07.915086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.915186 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.915275 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:02:07.917030 21230 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:07.918641 21230 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:07.925599 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.16.190.195:34143)
0927 21:02:07.921408 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.921413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.921649 (+ 236us) server_negotiation.cc:407] Connection header received
0927 21:02:07.921770 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.921773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.921790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.921845 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:07.922140 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923277 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.924518 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925149 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925239 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925300 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.925463 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.925505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260927 21:02:07.925588 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34143 (local address 127.0.0.1:34158)
0927 21:02:07.921585 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.921594 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.921676 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.921958 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.921959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.921960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:07.922043 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.922045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923391 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.923391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.924436 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.924437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925247 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.925330 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925345 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.925447 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.925505 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":6,"thread_start_us":91,"threads_started":1}
I20260927 21:02:07.925721 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:38519 (local address 127.0.0.1:55760)
0927 21:02:07.922167 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.922176 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.922224 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.922401 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.922402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.922404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:07.922441 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.922443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923590 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.923591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.924503 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.924506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925447 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.925524 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925528 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.925580 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.925627 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260927 21:02:07.925915 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.16.190.193:38519)
0927 21:02:07.921729 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.921734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.922241 (+ 507us) server_negotiation.cc:407] Connection header received
0927 21:02:07.922287 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.922290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.922305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.922389 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:07.922499 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923491 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.924576 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925282 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925356 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925407 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.925794 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.925836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260927 21:02:07.926183 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:33437 (local address 127.0.0.1:52840)
0927 21:02:07.921517 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.921561 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.921645 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.921954 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.921957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.921959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:07.922046 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.922047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923111 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.923112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.924417 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.924419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925445 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.925561 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925564 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.925609 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.925640 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":34,"thread_start_us":63,"threads_started":1}
I20260927 21:02:07.926596 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.921266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.16.190.194:33437)
0927 21:02:07.921477 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.921480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.921647 (+ 167us) server_negotiation.cc:407] Connection header received
0927 21:02:07.921777 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.921780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.921795 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.921865 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:07.922123 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.923002 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.924523 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.925282 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.925356 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.925407 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.926407 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.926499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260927 21:02:07.926666 21620 tablet_service.cc:1467] Processing CreateTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 (DEFAULT_TABLE table=test-table [id=a34850a3682c49159715e2ad16c6f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:07.926685 21354 tablet_service.cc:1467] Processing CreateTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 (DEFAULT_TABLE table=test-table [id=a34850a3682c49159715e2ad16c6f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:07.927019 21354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ced2f021f7f4f0c994711a6dba50981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:07.927023 21620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ced2f021f7f4f0c994711a6dba50981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:07.927206 21487 tablet_service.cc:1467] Processing CreateTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 (DEFAULT_TABLE table=test-table [id=a34850a3682c49159715e2ad16c6f4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:07.927484 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ced2f021f7f4f0c994711a6dba50981. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:07.929798 21703 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: Bootstrap starting.
I20260927 21:02:07.930394 21704 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Bootstrap starting.
I20260927 21:02:07.930723 21705 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Bootstrap starting.
I20260927 21:02:07.931082 21704 tablet_bootstrap.cc:654] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:07.931082 21703 tablet_bootstrap.cc:654] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:07.931294 21705 tablet_bootstrap.cc:654] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:07.931761 21704 log.cc:824] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:07.931779 21705 log.cc:824] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:07.931784 21703 log.cc:824] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:07.932564 21705 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: No bootstrap required, opened a new log
I20260927 21:02:07.932585 21704 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: No bootstrap required, opened a new log
I20260927 21:02:07.932654 21705 ts_tablet_manager.cc:1397] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:07.932659 21704 ts_tablet_manager.cc:1397] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:07.933193 21703 tablet_bootstrap.cc:492] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: No bootstrap required, opened a new log
I20260927 21:02:07.933264 21703 ts_tablet_manager.cc:1397] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:02:07.934227 21705 raft_consensus.cc:348] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.934389 21705 raft_consensus.cc:374] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:07.934434 21705 raft_consensus.cc:729] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1dc5ca38a74eaca254b55616a258fb, State: Initialized, Role: FOLLOWER
I20260927 21:02:07.934554 21705 consensus_queue.cc:260] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.934721 21704 raft_consensus.cc:348] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.934882 21704 raft_consensus.cc:374] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:07.934888 21705 ts_tablet_manager.cc:1428] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:07.934932 21704 raft_consensus.cc:729] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b84e4b906724760884f2ddeaa695992, State: Initialized, Role: FOLLOWER
I20260927 21:02:07.934885 21553 heartbeater.cc:503] Master 127.16.190.254:45379 was elected leader, sending a full tablet report...
I20260927 21:02:07.935038 21704 consensus_queue.cc:260] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.935303 21704 ts_tablet_manager.cc:1428] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:02:07.935441 21686 heartbeater.cc:503] Master 127.16.190.254:45379 was elected leader, sending a full tablet report...
I20260927 21:02:07.935357 21703 raft_consensus.cc:348] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.935491 21703 raft_consensus.cc:374] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:07.935519 21703 raft_consensus.cc:729] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ca2824d2884632bb4120cb05953a15, State: Initialized, Role: FOLLOWER
I20260927 21:02:07.935612 21703 consensus_queue.cc:260] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.935876 21420 heartbeater.cc:503] Master 127.16.190.254:45379 was elected leader, sending a full tablet report...
I20260927 21:02:07.936189 21703 ts_tablet_manager.cc:1428] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260927 21:02:07.975687 21554 tablet.cc:2369] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:07.989234 21711 raft_consensus.cc:482] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:07.989398 21711 raft_consensus.cc:504] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.989867 21711 leader_election.cc:288] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1dc5ca38a74eaca254b55616a258fb (127.16.190.194:33437), 9b84e4b906724760884f2ddeaa695992 (127.16.190.195:34143)
I20260927 21:02:07.993984 21422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.990028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:33437 (local address 127.16.190.193:38477)
0927 21:02:07.990205 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.990218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.990334 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.990556 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.990556 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.990557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:07.990658 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.990660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.991700 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.991702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.993075 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.993078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.993693 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.993788 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.993792 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.993843 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.993883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":73}
I20260927 21:02:07.994019 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.990245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:34771 (local address 127.16.190.195:34143)
0927 21:02:07.990416 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.990418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.990425 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:07.990448 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.990450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.990459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.990504 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:07.990743 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.991511 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993151 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.993567 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993654 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.993663 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.993847 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.993895 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260927 21:02:07.994241 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.990049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34143 (local address 127.16.190.193:34771)
0927 21:02:07.990266 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:07.990276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:07.990331 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:07.990555 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:07.990555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:07.990557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:07.990672 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.990674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.991620 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.991621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.993048 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:07.993051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.993661 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:07.993750 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.993756 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:02:07.994108 (+ 352us) client_negotiation.cc:240] Negotiation successful
0927 21:02:07.994157 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":25,"thread_start_us":85,"threads_started":1}
I20260927 21:02:07.994328 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:07.990132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:38477 (local address 127.16.190.194:33437)
0927 21:02:07.990190 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:07.990192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:07.990286 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:02:07.990443 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:07.990444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:07.990455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:07.990514 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:07.990740 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.991581 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993203 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:07.993605 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:07.993696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:07.993704 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:07.994216 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 21:02:07.994257 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:02:07.994668 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ced2f021f7f4f0c994711a6dba50981" candidate_uuid: "72ca2824d2884632bb4120cb05953a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b84e4b906724760884f2ddeaa695992" is_pre_election: true
I20260927 21:02:07.994705 21507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ced2f021f7f4f0c994711a6dba50981" candidate_uuid: "72ca2824d2884632bb4120cb05953a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1dc5ca38a74eaca254b55616a258fb" is_pre_election: true
I20260927 21:02:07.994834 21507 raft_consensus.cc:2454] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ca2824d2884632bb4120cb05953a15 in term 0.
I20260927 21:02:07.994834 21640 raft_consensus.cc:2454] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ca2824d2884632bb4120cb05953a15 in term 0.
I20260927 21:02:07.995105 21308 leader_election.cc:302] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ca2824d2884632bb4120cb05953a15, fa1dc5ca38a74eaca254b55616a258fb; no voters:
I20260927 21:02:07.995225 21711 raft_consensus.cc:2781] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:07.995275 21711 raft_consensus.cc:482] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:07.995297 21711 raft_consensus.cc:3037] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:07.996273 21711 raft_consensus.cc:504] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.996428 21711 leader_election.cc:288] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [CANDIDATE]: Term 1 election: Requested vote from peers fa1dc5ca38a74eaca254b55616a258fb (127.16.190.194:33437), 9b84e4b906724760884f2ddeaa695992 (127.16.190.195:34143)
I20260927 21:02:07.996608 21640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ced2f021f7f4f0c994711a6dba50981" candidate_uuid: "72ca2824d2884632bb4120cb05953a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b84e4b906724760884f2ddeaa695992"
I20260927 21:02:07.996618 21507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ced2f021f7f4f0c994711a6dba50981" candidate_uuid: "72ca2824d2884632bb4120cb05953a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1dc5ca38a74eaca254b55616a258fb"
I20260927 21:02:07.996690 21640 raft_consensus.cc:3037] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:07.996690 21507 raft_consensus.cc:3037] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:07.997648 21640 raft_consensus.cc:2454] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ca2824d2884632bb4120cb05953a15 in term 1.
I20260927 21:02:07.997648 21507 raft_consensus.cc:2454] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ca2824d2884632bb4120cb05953a15 in term 1.
I20260927 21:02:07.997799 21308 leader_election.cc:302] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ca2824d2884632bb4120cb05953a15, fa1dc5ca38a74eaca254b55616a258fb; no voters:
I20260927 21:02:07.997901 21711 raft_consensus.cc:2781] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:07.998051 21711 raft_consensus.cc:686] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 1 LEADER]: Becoming Leader. State: Replica: 72ca2824d2884632bb4120cb05953a15, State: Running, Role: LEADER
I20260927 21:02:07.998165 21711 consensus_queue.cc:237] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } }
I20260927 21:02:07.999073 21229 catalog_manager.cc:5495] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72ca2824d2884632bb4120cb05953a15 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "72ca2824d2884632bb4120cb05953a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ca2824d2884632bb4120cb05953a15" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 38519 } health_report { overall_health: HEALTHY } } }
I20260927 21:02:08.019352 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.016784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.16.190.193:38519)
0927 21:02:08.016858 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.016860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.016870 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:08.016959 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.016960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.016979 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.017077 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:08.017247 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:08.017742 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:08.018531 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:08.018727 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:08.018806 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:08.018992 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:08.019112 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:08.019215 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.019233 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.019253 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:02:08.021396 21507 raft_consensus.cc:1261] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 1 FOLLOWER]: Refusing update from remote peer 72ca2824d2884632bb4120cb05953a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:08.021399 21640 raft_consensus.cc:1261] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 1 FOLLOWER]: Refusing update from remote peer 72ca2824d2884632bb4120cb05953a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:08.021799 21713 consensus_queue.cc:1035] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1dc5ca38a74eaca254b55616a258fb" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:08.021948 21711 consensus_queue.cc:1035] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b84e4b906724760884f2ddeaa695992" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:08.024860 21717 mvcc.cc:205] Tried to move back new op lower bound from 7334063833172254720 to 7334063833081102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334063833172254720 or (T in {7334063833172254720})}]
I20260927 21:02:08.025703 21716 mvcc.cc:205] Tried to move back new op lower bound from 7334063833172254720 to 7334063833081102336. Current Snapshot: MvccSnapshot[applied={T|T < 7334063833172254720}]
I20260927 21:02:08.028836 21229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:08.028946 21229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:08.029989 21229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 663482b25c3f402aad8820f8712d4d2c: Sending DeleteTablet for 3 replicas of tablet 0ced2f021f7f4f0c994711a6dba50981
I20260927 21:02:08.030390 21487 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52840
I20260927 21:02:08.030396 21354 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:55760
I20260927 21:02:08.030435 21620 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ced2f021f7f4f0c994711a6dba50981 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:34158
I20260927 21:02:08.030647 21727 tablet_replica.cc:330] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15: stopping tablet replica
I20260927 21:02:08.030740 21727 raft_consensus.cc:2229] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:08.030864 21727 raft_consensus.cc:2258] T 0ced2f021f7f4f0c994711a6dba50981 P 72ca2824d2884632bb4120cb05953a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:08.030970 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21292
I20260927 21:02:08.031036 21725 tablet_replica.cc:330] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: stopping tablet replica
I20260927 21:02:08.032231 21725 raft_consensus.cc:2229] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:08.032343 21725 raft_consensus.cc:2258] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:08.033124 21725 ts_tablet_manager.cc:1905] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:08.035148 21725 ts_tablet_manager.cc:1918] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:08.035230 21725 log.cc:1196] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/0ced2f021f7f4f0c994711a6dba50981
I20260927 21:02:08.035494 21725 ts_tablet_manager.cc:1939] T 0ced2f021f7f4f0c994711a6dba50981 P fa1dc5ca38a74eaca254b55616a258fb: Deleting consensus metadata
I20260927 21:02:08.036026 21215 catalog_manager.cc:4841] TS fa1dc5ca38a74eaca254b55616a258fb (127.16.190.194:33437): tablet 0ced2f021f7f4f0c994711a6dba50981 (table test-table [id=a34850a3682c49159715e2ad16c6f4b0]) successfully deleted
W20260927 21:02:08.037523 21214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:38519 (error 108)
I20260927 21:02:08.037853 21726 tablet_replica.cc:330] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: stopping tablet replica
I20260927 21:02:08.037873 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21425
I20260927 21:02:08.038442 21726 raft_consensus.cc:2229] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:08.038473 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.037973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:38519 (local address 127.0.0.1:55790)
0927 21:02:08.038402 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.038446 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:38519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
I20260927 21:02:08.038579 21726 raft_consensus.cc:2258] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 21:02:08.038589 21214 catalog_manager.cc:4584] TS 72ca2824d2884632bb4120cb05953a15 (127.16.190.193:38519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ced2f021f7f4f0c994711a6dba50981: Network error: Client connection negotiation failed: client connection to 127.16.190.193:38519: connect: Connection refused (error 111)
I20260927 21:02:08.039772 21726 ts_tablet_manager.cc:1905] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:08.041785 21726 ts_tablet_manager.cc:1918] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:08.041872 21726 log.cc:1196] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/0ced2f021f7f4f0c994711a6dba50981
I20260927 21:02:08.042152 21726 ts_tablet_manager.cc:1939] T 0ced2f021f7f4f0c994711a6dba50981 P 9b84e4b906724760884f2ddeaa695992: Deleting consensus metadata
I20260927 21:02:08.042896 21214 catalog_manager.cc:4841] TS 9b84e4b906724760884f2ddeaa695992 (127.16.190.195:34143): tablet 0ced2f021f7f4f0c994711a6dba50981 (table test-table [id=a34850a3682c49159715e2ad16c6f4b0]) successfully deleted
I20260927 21:02:08.044806 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21558
I20260927 21:02:08.050001 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21199
2026-09-27T21:02:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2009 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T21:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:08Z Disabled control of system clock
I20260927 21:02:08.078805 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40515
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:40515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:02:08.159655 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:08.159893 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:08.159914 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:08.161158 21732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:08.161209 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:08.161223 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:08.161237 21732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:08.161294 21732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:08.162673 21732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:40515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:08.162953 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:08.163283 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:08.168375 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.168361 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:08.168583 21732 server_base.cc:1034] running on GCE node
W20260927 21:02:08.168355 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:08.168979 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:08.169275 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:08.170454 21732 hybrid_clock.cc:648] HybridClock initialized: now 1790542928170452 us; error 43 us; skew 500 ppm
I20260927 21:02:08.171819 21732 webserver.cc:460] Webserver started at http://127.16.190.254:44791/ using document root <none> and password file <none>
I20260927 21:02:08.172065 21732 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:08.172135 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:08.172303 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:08.173472 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "9800fbc1d4924b46961f93d878535511"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "e2ae25f461c50a08357b7cdde59fc6ab"
server_key_iv: "1e1a85112a2ab54968a515fa1c67e331"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.173851 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "9800fbc1d4924b46961f93d878535511"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "e2ae25f461c50a08357b7cdde59fc6ab"
server_key_iv: "1e1a85112a2ab54968a515fa1c67e331"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.175242 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:08.176069 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.176250 21732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:08.176358 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "9800fbc1d4924b46961f93d878535511"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "e2ae25f461c50a08357b7cdde59fc6ab"
server_key_iv: "1e1a85112a2ab54968a515fa1c67e331"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.176438 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:08.179973 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:08.180250 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:08.180395 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:08.185021 21732 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:40515
I20260927 21:02:08.185062 21798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:40515 every 8 connection(s)
I20260927 21:02:08.185449 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:08.186156 21799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:08.189074 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: Bootstrap starting.
I20260927 21:02:08.189723 21799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:08.190021 21799 log.cc:824] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:08.190750 21799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: No bootstrap required, opened a new log
I20260927 21:02:08.192328 21799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } }
I20260927 21:02:08.192497 21799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:08.192546 21799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800fbc1d4924b46961f93d878535511, State: Initialized, Role: FOLLOWER
I20260927 21:02:08.192747 21799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } }
I20260927 21:02:08.192893 21799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:08.192937 21799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:08.192987 21799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:08.193140 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21732
I20260927 21:02:08.193264 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:08.193451 17147 external_mini_cluster.cc:1428] Setting key c8840fde4bef20221f5156f7cfb5ec81
I20260927 21:02:08.195834 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.194097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.16.190.254:40515)
0927 21:02:08.194340 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.194348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.194375 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:02:08.194473 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.194479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.194515 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.194633 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.195363 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.195366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.195366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.195369 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.195401 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.195412 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.195466 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.195547 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.195627 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":127,"threads_started":1}
I20260927 21:02:08.226279 21799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } }
I20260927 21:02:08.226498 21799 leader_election.cc:302] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9800fbc1d4924b46961f93d878535511; no voters:
I20260927 21:02:08.226743 21799 leader_election.cc:288] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:08.226825 21804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:08.227008 21804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [term 1 LEADER]: Becoming Leader. State: Replica: 9800fbc1d4924b46961f93d878535511, State: Running, Role: LEADER
I20260927 21:02:08.227047 21799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:08.227131 21804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } }
I20260927 21:02:08.227576 21805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9800fbc1d4924b46961f93d878535511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } } }
I20260927 21:02:08.227684 21805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:08.228193 21806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9800fbc1d4924b46961f93d878535511. Latest consensus state: current_term: 1 leader_uuid: "9800fbc1d4924b46961f93d878535511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9800fbc1d4924b46961f93d878535511" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40515 } } }
I20260927 21:02:08.228200 21813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:08.228358 21806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:08.228914 21813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:08.230873 21813 catalog_manager.cc:1347] Generated new cluster ID: 950d920ed7bd4d9288c445353bfd598f
I20260927 21:02:08.230933 21813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:08.261286 21813 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:08.261510 21813 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:08.266799 21813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: Generated new TSK 0
I20260927 21:02:08.267041 21813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:08.270653 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--builtin_ntp_servers=127.16.190.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 21:02:08.358282 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:08.358510 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:08.358534 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:08.359706 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:08.359747 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:08.359774 21823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:08.361227 21823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:08.361449 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:08.361716 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:08.366986 21823 server_base.cc:1034] running on GCE node
W20260927 21:02:08.367105 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.367223 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.367137 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:08.367487 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:08.367754 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:08.368870 21823 hybrid_clock.cc:648] HybridClock initialized: now 1790542928368857 us; error 27 us; skew 500 ppm
I20260927 21:02:08.370231 21823 webserver.cc:460] Webserver started at http://127.16.190.193:33233/ using document root <none> and password file <none>
I20260927 21:02:08.370500 21823 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:08.370548 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:08.370695 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:08.371819 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "60649bf1c3cf474783bd86fd73e2aa30"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "8c6481cbfde98b63059e47b9f1f236fa"
server_key_iv: "d410ce401ba3e1a21d3f17c40899176d"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.372182 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "60649bf1c3cf474783bd86fd73e2aa30"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "8c6481cbfde98b63059e47b9f1f236fa"
server_key_iv: "d410ce401ba3e1a21d3f17c40899176d"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.373559 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:08.374332 21837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.374501 21823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:08.374604 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "60649bf1c3cf474783bd86fd73e2aa30"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "8c6481cbfde98b63059e47b9f1f236fa"
server_key_iv: "d410ce401ba3e1a21d3f17c40899176d"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.374680 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:08.378064 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:08.378343 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:08.378484 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:08.378741 21823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:08.379050 21823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:08.379107 21823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.379151 21823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:08.379172 21823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.384922 21823 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42599
I20260927 21:02:08.385011 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42599 every 8 connection(s)
I20260927 21:02:08.385361 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:08.385673 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21823
I20260927 21:02:08.385774 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:08.385963 17147 external_mini_cluster.cc:1428] Setting key a64eabe1d7c3a1492fb46d93dbd81cd0
I20260927 21:02:08.387053 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--builtin_ntp_servers=127.16.190.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:02:08.389236 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.386395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:40125 (local address 127.16.190.254:40515)
0927 21:02:08.386446 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.386448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.386786 (+ 338us) server_negotiation.cc:407] Connection header received
0927 21:02:08.387374 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.387375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.387383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.387430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.388989 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.388990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.388990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.388994 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.389011 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.389015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.389073 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.389166 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.389180 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:08.389487 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.386385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40515 (local address 127.16.190.193:40125)
0927 21:02:08.386631 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.386679 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.387261 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.387505 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.387516 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.387527 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.388771 (+ 1244us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.388772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.388847 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.388890 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.389087 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.389091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.389095 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.389179 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.389248 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260927 21:02:08.390067 21951 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40515
I20260927 21:02:08.390183 21951 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:08.390336 21951 heartbeater.cc:511] Master 127.16.190.254:40515 requested a full tablet report, sending...
I20260927 21:02:08.390918 21763 ts_manager.cc:194] Registered new tserver with Master: 60649bf1c3cf474783bd86fd73e2aa30 (127.16.190.193:42599)
W20260927 21:02:08.470705 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:08.470937 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:08.470963 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:08.472215 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:08.472263 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:08.472292 21954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:08.473733 21954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:08.473965 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:08.474239 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:08.479729 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.479750 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.479736 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:08.480185 21954 server_base.cc:1034] running on GCE node
I20260927 21:02:08.480439 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:08.480757 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:08.481918 21954 hybrid_clock.cc:648] HybridClock initialized: now 1790542928481902 us; error 29 us; skew 500 ppm
I20260927 21:02:08.483430 21954 webserver.cc:460] Webserver started at http://127.16.190.194:44905/ using document root <none> and password file <none>
I20260927 21:02:08.483693 21954 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:08.483747 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:08.483883 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:08.485121 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "b19924fdaff546a48678138c0fc6f00a"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "7a6515b6ee0ff226fe50a14ce85b5622"
server_key_iv: "9d1c35a77fd219c4fed2da8755782a0a"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.485526 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "b19924fdaff546a48678138c0fc6f00a"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "7a6515b6ee0ff226fe50a14ce85b5622"
server_key_iv: "9d1c35a77fd219c4fed2da8755782a0a"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.486934 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:08.487766 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.487947 21954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:08.488046 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "b19924fdaff546a48678138c0fc6f00a"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "7a6515b6ee0ff226fe50a14ce85b5622"
server_key_iv: "9d1c35a77fd219c4fed2da8755782a0a"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.488142 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:08.491796 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:08.492077 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:08.492218 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:08.492429 21954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:08.492753 21954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:08.492805 21954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.492863 21954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:08.492892 21954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.499037 21954 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:43759
I20260927 21:02:08.499132 22081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:43759 every 8 connection(s)
I20260927 21:02:08.499491 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:08.502650 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 21954
I20260927 21:02:08.502749 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:08.502945 17147 external_mini_cluster.cc:1428] Setting key 504f3f9cc425d80cd47a8b66c2717c08
I20260927 21:02:08.503269 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.500418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:53719 (local address 127.16.190.254:40515)
0927 21:02:08.500489 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.500492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.500764 (+ 272us) server_negotiation.cc:407] Connection header received
0927 21:02:08.501410 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.501411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.501418 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.501471 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.502948 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.502951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.502951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.502955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.502972 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.502975 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.503033 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.503162 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.503182 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:08.503463 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.500338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40515 (local address 127.16.190.194:53719)
0927 21:02:08.500606 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.500660 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.501289 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.501549 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.501553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.501570 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.502681 (+ 1111us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.502681 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.502793 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.502835 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.503055 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.503062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.503065 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.503151 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.503222 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260927 21:02:08.503926 22082 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40515
I20260927 21:02:08.504047 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--builtin_ntp_servers=127.16.190.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 21:02:08.504062 22082 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:08.504249 22082 heartbeater.cc:511] Master 127.16.190.254:40515 requested a full tablet report, sending...
I20260927 21:02:08.504715 21763 ts_manager.cc:194] Registered new tserver with Master: b19924fdaff546a48678138c0fc6f00a (127.16.190.194:43759)
W20260927 21:02:08.591351 22085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:08.591609 22085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:08.591646 22085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:08.592976 22085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:08.593077 22085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:08.593124 22085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:08.594646 22085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:08.594933 22085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:08.595259 22085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:08.600706 22090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.600703 22093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:08.600844 22091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:08.601303 22085 server_base.cc:1034] running on GCE node
I20260927 21:02:08.601557 22085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:08.601810 22085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:08.605058 22085 hybrid_clock.cc:648] HybridClock initialized: now 1790542928605031 us; error 39 us; skew 500 ppm
I20260927 21:02:08.606392 22085 webserver.cc:460] Webserver started at http://127.16.190.195:33443/ using document root <none> and password file <none>
I20260927 21:02:08.606643 22085 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:08.606738 22085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:08.606899 22085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:08.607991 22085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "2639344df917490a8f52b1cc110de947"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "cc3d5e6e3c21333b41dce8b6cef456e9"
server_key_iv: "04011b95aaf4a0d44d73a5e5141a61c2"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.608369 22085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "2639344df917490a8f52b1cc110de947"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "cc3d5e6e3c21333b41dce8b6cef456e9"
server_key_iv: "04011b95aaf4a0d44d73a5e5141a61c2"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.609869 22085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:08.610885 22099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.611146 22085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:08.611260 22085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "2639344df917490a8f52b1cc110de947"
format_stamp: "Formatted at 2026-09-27 21:02:08 on dist-test-slave-6c7s"
server_key: "cc3d5e6e3c21333b41dce8b6cef456e9"
server_key_iv: "04011b95aaf4a0d44d73a5e5141a61c2"
server_key_version: "encryptionkey@0"
I20260927 21:02:08.611364 22085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:08.615015 22085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:08.615329 22085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:08.615486 22085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:08.615710 22085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:08.616042 22085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:08.616098 22085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.616132 22085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:08.616159 22085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:08.622025 22085 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:45373
I20260927 21:02:08.622105 22212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:45373 every 8 connection(s)
I20260927 21:02:08.622440 22085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:08.625691 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.623224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:47483 (local address 127.16.190.254:40515)
0927 21:02:08.623307 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.623310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.623515 (+ 205us) server_negotiation.cc:407] Connection header received
0927 21:02:08.624011 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.624013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.624018 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.624078 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.625424 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.625425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.625425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.625431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.625454 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.625459 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.625518 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.625613 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.625636 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:02:08.625833 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.623185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40515 (local address 127.16.190.195:47483)
0927 21:02:08.623397 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.623431 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.623901 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.624158 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.624162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.624170 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.625266 (+ 1096us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.625266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.625296 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.625326 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.625527 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.625531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.625534 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.625594 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.625642 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260927 21:02:08.626346 22213 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40515
I20260927 21:02:08.626426 22213 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:08.626591 22213 heartbeater.cc:511] Master 127.16.190.254:40515 requested a full tablet report, sending...
I20260927 21:02:08.627002 21763 ts_manager.cc:194] Registered new tserver with Master: 2639344df917490a8f52b1cc110de947 (127.16.190.195:45373)
I20260927 21:02:08.629267 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22085
I20260927 21:02:08.629380 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:08.629549 17147 external_mini_cluster.cc:1428] Setting key e6177444160b19116bf6c29ce4de7cc3
I20260927 21:02:08.630345 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:02:08.632673 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.631647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.16.190.254:40515)
0927 21:02:08.631728 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.631730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.631779 (+ 49us) server_negotiation.cc:407] Connection header received
0927 21:02:08.631858 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.631859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.631863 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.631921 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.632457 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.632458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.632458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.632462 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.632477 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.632481 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.632522 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.632588 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.632602 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:02:08.634334 21763 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:08.635995 21763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:08.639923 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42599 (local address 127.0.0.1:40098)
0927 21:02:08.638638 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.638677 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.638821 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.638961 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.638965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.638972 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.639497 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.639498 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.639515 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.639540 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.639748 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.639753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.639756 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.639825 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.639839 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":70,"threads_started":1}
I20260927 21:02:08.639947 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.16.190.193:42599)
0927 21:02:08.638590 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.638595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.638791 (+ 196us) server_negotiation.cc:407] Connection header received
0927 21:02:08.638880 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.638884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.638894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.638962 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.639602 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.639608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.639608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.639612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.639649 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.639666 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.639722 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.639857 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.639874 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260927 21:02:08.640478 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:45373 (local address 127.0.0.1:38890)
0927 21:02:08.638785 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.638796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.638855 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.639484 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.639485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.639489 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.640114 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.640115 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.640121 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.640132 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.640354 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.640355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.640355 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.640403 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.640418 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":93,"threads_started":1}
I20260927 21:02:08.640497 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.16.190.195:45373)
0927 21:02:08.638806 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.638813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.638846 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:02:08.638923 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.638926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.638937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.639007 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.640223 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.640230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.640230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.640234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.640266 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.640287 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.640346 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.640406 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.640421 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260927 21:02:08.640780 21885 tablet_service.cc:1467] Processing CreateTablet for tablet 5732c96915954549a82eb796105e70db (DEFAULT_TABLE table=test-table [id=80d7ae35082e4ca7a563d70c6677477b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:08.641163 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.16.190.194:43759)
0927 21:02:08.638775 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.638780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.639163 (+ 383us) server_negotiation.cc:407] Connection header received
0927 21:02:08.639194 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.639197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.639206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.639271 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.640437 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.640443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.640444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.640448 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.640479 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.640506 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.640849 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.641068 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.641085 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260927 21:02:08.641141 22147 tablet_service.cc:1467] Processing CreateTablet for tablet 5732c96915954549a82eb796105e70db (DEFAULT_TABLE table=test-table [id=80d7ae35082e4ca7a563d70c6677477b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:08.641227 21885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5732c96915954549a82eb796105e70db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:08.641384 22147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5732c96915954549a82eb796105e70db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:08.641392 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.638365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:43759 (local address 127.0.0.1:36100)
0927 21:02:08.639031 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.639041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.639105 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.639577 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.639578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.639581 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.640341 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.640341 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.640347 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.640359 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.640877 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.640878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.640878 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.640925 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.640939 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260927 21:02:08.642241 22016 tablet_service.cc:1467] Processing CreateTablet for tablet 5732c96915954549a82eb796105e70db (DEFAULT_TABLE table=test-table [id=80d7ae35082e4ca7a563d70c6677477b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:08.642553 22016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5732c96915954549a82eb796105e70db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:08.643832 22227 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Bootstrap starting.
I20260927 21:02:08.644575 22227 tablet_bootstrap.cc:654] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:08.644558 22228 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Bootstrap starting.
I20260927 21:02:08.644953 22227 log.cc:824] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:08.645232 22228 tablet_bootstrap.cc:654] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:08.645572 22228 log.cc:824] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:08.645783 22227 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: No bootstrap required, opened a new log
I20260927 21:02:08.645851 22227 ts_tablet_manager.cc:1397] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:02:08.645877 22230 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Bootstrap starting.
I20260927 21:02:08.646489 22228 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: No bootstrap required, opened a new log
I20260927 21:02:08.646509 22230 tablet_bootstrap.cc:654] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:08.646615 22228 ts_tablet_manager.cc:1397] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:02:08.646812 22230 log.cc:824] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:08.647456 22230 tablet_bootstrap.cc:492] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: No bootstrap required, opened a new log
I20260927 21:02:08.647533 22230 ts_tablet_manager.cc:1397] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:08.648514 22227 raft_consensus.cc:348] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.648738 22227 raft_consensus.cc:374] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:08.648780 22227 raft_consensus.cc:729] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2639344df917490a8f52b1cc110de947, State: Initialized, Role: FOLLOWER
I20260927 21:02:08.648967 22227 consensus_queue.cc:260] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.649232 22227 ts_tablet_manager.cc:1428] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 21:02:08.649338 22213 heartbeater.cc:503] Master 127.16.190.254:40515 was elected leader, sending a full tablet report...
I20260927 21:02:08.649223 22228 raft_consensus.cc:348] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.649367 22228 raft_consensus.cc:374] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:08.649398 22228 raft_consensus.cc:729] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60649bf1c3cf474783bd86fd73e2aa30, State: Initialized, Role: FOLLOWER
I20260927 21:02:08.649497 22228 consensus_queue.cc:260] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.649773 22228 ts_tablet_manager.cc:1428] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:02:08.649806 21951 heartbeater.cc:503] Master 127.16.190.254:40515 was elected leader, sending a full tablet report...
I20260927 21:02:08.649662 22230 raft_consensus.cc:348] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.650107 22230 raft_consensus.cc:374] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:08.650153 22230 raft_consensus.cc:729] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19924fdaff546a48678138c0fc6f00a, State: Initialized, Role: FOLLOWER
I20260927 21:02:08.650281 22230 consensus_queue.cc:260] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.650537 22230 ts_tablet_manager.cc:1428] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:02:08.650701 22082 heartbeater.cc:503] Master 127.16.190.254:40515 was elected leader, sending a full tablet report...
W20260927 21:02:08.750102 22083 tablet.cc:2369] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:08.835740 22233 raft_consensus.cc:482] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:08.835904 22233 raft_consensus.cc:504] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.836362 22233 leader_election.cc:288] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60649bf1c3cf474783bd86fd73e2aa30 (127.16.190.193:42599), b19924fdaff546a48678138c0fc6f00a (127.16.190.194:43759)
I20260927 21:02:08.837843 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.836683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:55751 (local address 127.16.190.194:43759)
0927 21:02:08.836727 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.836729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.836741 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:08.836829 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.836830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.836836 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.836910 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.837533 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.837535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.837535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.837540 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.837557 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.837561 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.837664 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.837758 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.837773 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:08.837842 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.836603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:43759 (local address 127.16.190.195:55751)
0927 21:02:08.836668 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.836682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.836742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.836928 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.836929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.836931 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.837390 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.837391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.837397 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.837407 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.837664 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.837665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.837666 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.837748 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.837762 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:02:08.837873 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.836693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:49493 (local address 127.16.190.193:42599)
0927 21:02:08.836747 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.836748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.836976 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:02:08.837013 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.837014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.837019 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.837062 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.837644 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.837646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.837646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.837651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.837669 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.837674 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.837735 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.837795 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.837808 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:02:08.837862 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.836638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42599 (local address 127.16.190.195:49493)
0927 21:02:08.836886 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.836896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:08.836947 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:08.837092 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:08.837093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:08.837095 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:08.837539 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:08.837539 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:08.837544 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:08.837555 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:08.837740 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:08.837741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:08.837742 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 21:02:08.837787 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 21:02:08.837797 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":187,"threads_started":1}
I20260927 21:02:08.838197 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5732c96915954549a82eb796105e70db" candidate_uuid: "2639344df917490a8f52b1cc110de947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19924fdaff546a48678138c0fc6f00a" is_pre_election: true
I20260927 21:02:08.838356 22036 raft_consensus.cc:2454] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2639344df917490a8f52b1cc110de947 in term 0.
I20260927 21:02:08.838379 21905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5732c96915954549a82eb796105e70db" candidate_uuid: "2639344df917490a8f52b1cc110de947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60649bf1c3cf474783bd86fd73e2aa30" is_pre_election: true
I20260927 21:02:08.838564 21905 raft_consensus.cc:2454] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2639344df917490a8f52b1cc110de947 in term 0.
I20260927 21:02:08.838620 22103 leader_election.cc:302] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2639344df917490a8f52b1cc110de947, b19924fdaff546a48678138c0fc6f00a; no voters:
I20260927 21:02:08.838753 22233 raft_consensus.cc:2781] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:08.838851 22233 raft_consensus.cc:482] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:08.838899 22233 raft_consensus.cc:3037] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 0 FOLLOWER]: Advancing to term 1
W20260927 21:02:08.873037 22214 tablet.cc:2369] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:02:08.886291 21952 tablet.cc:2369] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:08.896875 22233 raft_consensus.cc:504] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.897125 22233 leader_election.cc:288] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [CANDIDATE]: Term 1 election: Requested vote from peers 60649bf1c3cf474783bd86fd73e2aa30 (127.16.190.193:42599), b19924fdaff546a48678138c0fc6f00a (127.16.190.194:43759)
I20260927 21:02:08.897436 21905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5732c96915954549a82eb796105e70db" candidate_uuid: "2639344df917490a8f52b1cc110de947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60649bf1c3cf474783bd86fd73e2aa30"
I20260927 21:02:08.897559 21905 raft_consensus.cc:3037] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:08.897446 22036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5732c96915954549a82eb796105e70db" candidate_uuid: "2639344df917490a8f52b1cc110de947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19924fdaff546a48678138c0fc6f00a"
I20260927 21:02:08.897763 22036 raft_consensus.cc:3037] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:08.898608 22036 raft_consensus.cc:2454] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2639344df917490a8f52b1cc110de947 in term 1.
I20260927 21:02:08.898712 21905 raft_consensus.cc:2454] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2639344df917490a8f52b1cc110de947 in term 1.
I20260927 21:02:08.898833 22103 leader_election.cc:302] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2639344df917490a8f52b1cc110de947, b19924fdaff546a48678138c0fc6f00a; no voters:
I20260927 21:02:08.898975 22233 raft_consensus.cc:2781] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:08.899156 22233 raft_consensus.cc:686] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 1 LEADER]: Becoming Leader. State: Replica: 2639344df917490a8f52b1cc110de947, State: Running, Role: LEADER
I20260927 21:02:08.899268 22233 consensus_queue.cc:237] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } }
I20260927 21:02:08.900287 21763 catalog_manager.cc:5495] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2639344df917490a8f52b1cc110de947 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "2639344df917490a8f52b1cc110de947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2639344df917490a8f52b1cc110de947" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 45373 } health_report { overall_health: HEALTHY } } }
I20260927 21:02:08.928404 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.927228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.16.190.195:45373)
0927 21:02:08.927296 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:08.927298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:08.927309 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:02:08.927375 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:08.927376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:08.927400 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:08.927473 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:08.928144 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:08.928146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:08.928146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:08.928151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:08.928173 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:08.928178 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:08.928258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:08.928307 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 21:02:08.928323 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:02:08.930306 21905 raft_consensus.cc:1261] T 5732c96915954549a82eb796105e70db P 60649bf1c3cf474783bd86fd73e2aa30 [term 1 FOLLOWER]: Refusing update from remote peer 2639344df917490a8f52b1cc110de947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:08.930341 22036 raft_consensus.cc:1261] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 1 FOLLOWER]: Refusing update from remote peer 2639344df917490a8f52b1cc110de947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:08.930644 22233 consensus_queue.cc:1035] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60649bf1c3cf474783bd86fd73e2aa30" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:08.930786 22237 consensus_queue.cc:1035] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19924fdaff546a48678138c0fc6f00a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:08.933207 22240 mvcc.cc:205] Tried to move back new op lower bound from 7334063836895637504 to 7334063836772089856. Current Snapshot: MvccSnapshot[applied={T|T < 7334063836895637504}]
I20260927 21:02:08.933704 22241 mvcc.cc:205] Tried to move back new op lower bound from 7334063836895637504 to 7334063836772089856. Current Snapshot: MvccSnapshot[applied={T|T < 7334063836895637504}]
I20260927 21:02:08.933820 22242 mvcc.cc:205] Tried to move back new op lower bound from 7334063836895637504 to 7334063836772089856. Current Snapshot: MvccSnapshot[applied={T|T < 7334063836895637504 or (T in {7334063836895637504})}]
I20260927 21:02:08.935742 21763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:08.935840 21763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:08.936652 21763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9800fbc1d4924b46961f93d878535511: Sending DeleteTablet for 3 replicas of tablet 5732c96915954549a82eb796105e70db
I20260927 21:02:08.937073 21885 tablet_service.cc:1514] Processing DeleteTablet for tablet 5732c96915954549a82eb796105e70db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='slave'} at 127.0.0.1:40098
I20260927 21:02:08.937217 22016 tablet_service.cc:1514] Processing DeleteTablet for tablet 5732c96915954549a82eb796105e70db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='slave'} at 127.0.0.1:36100
I20260927 21:02:08.937297 22147 tablet_service.cc:1514] Processing DeleteTablet for tablet 5732c96915954549a82eb796105e70db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:08 UTC) from {username='slave'} at 127.0.0.1:38890
I20260927 21:02:08.937500 22252 tablet_replica.cc:330] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: stopping tablet replica
I20260927 21:02:08.937589 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21823
I20260927 21:02:08.937608 22252 raft_consensus.cc:2229] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:08.937743 22252 raft_consensus.cc:2258] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:08.938635 22252 ts_tablet_manager.cc:1905] T 5732c96915954549a82eb796105e70db P 2639344df917490a8f52b1cc110de947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:08.940444 22251 tablet_replica.cc:330] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: stopping tablet replica
I20260927 21:02:08.940584 22251 raft_consensus.cc:2229] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:08.940712 22251 raft_consensus.cc:2258] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:08.941260 22251 ts_tablet_manager.cc:1905] T 5732c96915954549a82eb796105e70db P b19924fdaff546a48678138c0fc6f00a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:02:08.944283 21748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:42599 (error 108)
I20260927 21:02:08.944914 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21954
I20260927 21:02:08.944947 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:08.944763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42599 (local address 127.0.0.1:40110)
0927 21:02:08.944860 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:08.944907 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 21:02:08.948608 21748 catalog_manager.cc:4584] TS 60649bf1c3cf474783bd86fd73e2aa30 (127.16.190.193:42599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5732c96915954549a82eb796105e70db: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42599: connect: Connection refused (error 111)
I20260927 21:02:09.001734 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.001574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42599 (local address 127.0.0.1:40120)
0927 21:02:09.001639 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.001691 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:02:09.003896 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.003756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42599 (local address 127.0.0.1:40128)
0927 21:02:09.003817 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.003846 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:02:09.006701 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.006577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:43759 (local address 127.0.0.1:36106)
0927 21:02:09.006626 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.006652 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.194:43759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 21:02:09.006693 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22085
I20260927 21:02:09.018954 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 21732
I20260927 21:02:09.019078 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.018925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:45373 (local address 127.0.0.1:38918)
0927 21:02:09.018997 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.019038 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.195:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
2026-09-27T21:02:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (968 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 21:02:09.039683 17147 mini_oidc.cc:152] Starting JWKS server
I20260927 21:02:09.040860 17147 webserver.cc:460] Webserver started at https://127.0.0.1:36485/ using document root <none> and password file <none>
I20260927 21:02:09.041005 17147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:02:09.041219 17147 webserver.cc:460] Webserver started at http://127.0.0.1:40729/ using document root <none> and password file <none>
2026-09-27T21:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:09Z Disabled control of system clock
I20260927 21:02:09.049870 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:34991
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:40819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:34991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/testchainca.cert with env {}
W20260927 21:02:09.133708 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:09.133925 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:09.133948 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:09.135170 22259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:09.135215 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:09.135228 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:09.135241 22259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:09.135252 22259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:02:09.135265 22259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:02:09.135277 22259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36485/jwks.json
I20260927 21:02:09.136795 22259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:40819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:34991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:09.137023 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:09.137184 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:09.142151 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.142280 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.142143 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:09.142449 22259 server_base.cc:1034] running on GCE node
I20260927 21:02:09.142704 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:09.143002 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:09.144227 22259 hybrid_clock.cc:648] HybridClock initialized: now 1790542929144204 us; error 38 us; skew 500 ppm
I20260927 21:02:09.145874 22259 webserver.cc:460] Webserver started at http://127.16.190.254:36505/ using document root <none> and password file <none>
I20260927 21:02:09.146291 22259 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:09.146358 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:09.146528 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:09.147717 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "85fa841362ae43f1b0995e7289423bfb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "e23552fcc13c25d5afae7efd65e29e34"
server_key_iv: "28b067d936750b53823508907ebf32a2"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.148118 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "85fa841362ae43f1b0995e7289423bfb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "e23552fcc13c25d5afae7efd65e29e34"
server_key_iv: "28b067d936750b53823508907ebf32a2"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.149734 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:09.150682 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:09.150943 22259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:09.151073 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "85fa841362ae43f1b0995e7289423bfb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "e23552fcc13c25d5afae7efd65e29e34"
server_key_iv: "28b067d936750b53823508907ebf32a2"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.151266 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:09.171353 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:09.171720 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:09.171886 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:09.176369 22259 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:34991
I20260927 21:02:09.176405 22325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:34991 every 8 connection(s)
I20260927 21:02:09.176793 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:09.177976 22326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:09.180464 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: Bootstrap starting.
I20260927 21:02:09.181278 22326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:09.181610 22326 log.cc:824] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:09.182425 22326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: No bootstrap required, opened a new log
I20260927 21:02:09.184227 22326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } }
I20260927 21:02:09.184391 22326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:09.184417 22326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85fa841362ae43f1b0995e7289423bfb, State: Initialized, Role: FOLLOWER
I20260927 21:02:09.184582 22326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } }
I20260927 21:02:09.184661 22326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:09.184692 22326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:09.184751 22326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:09.185335 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22259
I20260927 21:02:09.185422 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:09.185593 17147 external_mini_cluster.cc:1428] Setting key c81f78d6eb160fff858454d74fc8b41e
I20260927 21:02:09.190807 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.186127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.16.190.254:34991)
0927 21:02:09.186359 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.186367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.186398 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:02:09.186501 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.186507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.186562 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.186686 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:09.186879 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.188029 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.188898 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.189337 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.189438 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.190196 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:09.190199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:09.190199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:09.190374 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:09.190405 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:09.190416 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:09.190468 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:09.190559 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:02:09.190624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260927 21:02:09.191910 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:02:09.202224 22326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } }
I20260927 21:02:09.202426 22326 leader_election.cc:302] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85fa841362ae43f1b0995e7289423bfb; no voters:
I20260927 21:02:09.202656 22326 leader_election.cc:288] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:09.202773 22331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:09.202980 22331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [term 1 LEADER]: Becoming Leader. State: Replica: 85fa841362ae43f1b0995e7289423bfb, State: Running, Role: LEADER
I20260927 21:02:09.203018 22326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:09.203131 22331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } }
I20260927 21:02:09.203523 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85fa841362ae43f1b0995e7289423bfb. Latest consensus state: current_term: 1 leader_uuid: "85fa841362ae43f1b0995e7289423bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } } }
I20260927 21:02:09.203619 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:09.203545 22332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85fa841362ae43f1b0995e7289423bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fa841362ae43f1b0995e7289423bfb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34991 } } }
I20260927 21:02:09.203773 22332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb [sys.catalog]: This master's current role is: LEADER
W20260927 21:02:09.205049 22346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:02:09.205119 22346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:02:09.205209 22338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:09.205431 22338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:09.207764 22338 catalog_manager.cc:1347] Generated new cluster ID: afd55a73f08c4cd1b5c9a45ac8b13b0e
I20260927 21:02:09.207837 22338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:09.234221 22338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:09.234824 22338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:09.239473 22338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85fa841362ae43f1b0995e7289423bfb: Generated new TSK 0
I20260927 21:02:09.239709 22338 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:02:09.264993 22356 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 21:02:09.265112 22356 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 21:02:09.265722 22355 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:09.248152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:34991 (local address 127.0.0.1:45236)
0927 21:02:09.248367 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.248378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:09.248442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:09.248732 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:09.248733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:09.248744 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:02:09.248805 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.248859 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.249923 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.249924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.250568 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.250570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.251119 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.251120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.251175 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.251197 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:02:09.265257 (+ 14060us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:02:09.265380 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.190.254:34991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":125,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.190.254:34991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 21:02:09.266068 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.248255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.16.190.254:34991)
0927 21:02:09.248311 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.248314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.248453 (+ 139us) server_negotiation.cc:407] Connection header received
0927 21:02:09.248524 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.248525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.248539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.248727 (+ 188us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:02:09.248956 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.249791 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.250666 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.250967 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.251043 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.251353 (+ 310us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:02:09.265080 (+ 13727us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 21:02:09.266004 (+ 924us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 21:02:09.266116 22330 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260927 21:02:09.266136 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22259
2026-09-27T21:02:09Z chronyd exiting
I20260927 21:02:09.302728 17147 test_util.cc:175] -----------------------------------------------
I20260927 21:02:09.302832 17147 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790542906842326-17147-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 21:02:09.303589 17147 mini_oidc.cc:152] Starting JWKS server
I20260927 21:02:09.304292 17147 webserver.cc:460] Webserver started at https://127.0.0.1:39539/ using document root <none> and password file <none>
I20260927 21:02:09.304361 17147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:02:09.304545 17147 webserver.cc:460] Webserver started at http://127.0.0.1:39229/ using document root <none> and password file <none>
2026-09-27T21:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:09Z Disabled control of system clock
I20260927 21:02:09.312512 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:43851
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:41997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:43851
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/testchainca.cert with env {}
W20260927 21:02:09.395763 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:09.395989 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:09.396013 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:09.397157 22362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:09.397193 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:09.397207 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:09.397218 22362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:09.397238 22362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:02:09.397250 22362 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:02:09.397261 22362 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39539/jwks.json
I20260927 21:02:09.398737 22362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:43851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:09.398983 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:09.399230 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:09.405498 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.405520 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.405498 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:09.405829 22362 server_base.cc:1034] running on GCE node
I20260927 21:02:09.406149 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:09.406483 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:09.407732 22362 hybrid_clock.cc:648] HybridClock initialized: now 1790542929407705 us; error 41 us; skew 500 ppm
I20260927 21:02:09.409246 22362 webserver.cc:460] Webserver started at http://127.16.190.254:39455/ using document root <none> and password file <none>
I20260927 21:02:09.409509 22362 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:09.409582 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:09.409718 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:09.410856 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "ee1ed9e54e7649db937b380a5d2459f0"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "26f000d2371c8f48cd7b2118eaf16478"
server_key_iv: "8224e26d6ef48f438060977798b04241"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.411244 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "ee1ed9e54e7649db937b380a5d2459f0"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "26f000d2371c8f48cd7b2118eaf16478"
server_key_iv: "8224e26d6ef48f438060977798b04241"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.412672 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:09.413669 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:09.413937 22362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:09.414054 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "ee1ed9e54e7649db937b380a5d2459f0"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "26f000d2371c8f48cd7b2118eaf16478"
server_key_iv: "8224e26d6ef48f438060977798b04241"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.414139 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:09.426430 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:09.426784 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:09.426942 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:09.431232 22362 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:43851
I20260927 21:02:09.431283 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:43851 every 8 connection(s)
I20260927 21:02:09.431677 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:09.432420 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:09.435554 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0: Bootstrap starting.
I20260927 21:02:09.436215 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:09.436547 22429 log.cc:824] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:09.437384 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22362
I20260927 21:02:09.437426 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0: No bootstrap required, opened a new log
I20260927 21:02:09.437529 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:09.437706 17147 external_mini_cluster.cc:1428] Setting key 0cda2af81d36a562e7510b32c0db4e52
I20260927 21:02:09.440217 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } }
I20260927 21:02:09.440428 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:09.440469 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1ed9e54e7649db937b380a5d2459f0, State: Initialized, Role: FOLLOWER
I20260927 21:02:09.440603 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } }
I20260927 21:02:09.440681 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:09.440712 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:09.440759 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:09.444083 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.438269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.16.190.254:43851)
0927 21:02:09.438473 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.438481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.438506 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:02:09.438910 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.438917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.438983 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.439097 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:09.439371 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.440232 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.442068 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.442504 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.442602 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.443404 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:09.443408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:09.443409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:09.443579 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:09.443612 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:09.443623 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:09.443676 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:09.443774 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 21:02:09.443887 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260927 21:02:09.445369 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:02:09.501094 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } }
I20260927 21:02:09.501314 22429 leader_election.cc:302] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee1ed9e54e7649db937b380a5d2459f0; no voters:
I20260927 21:02:09.501559 22429 leader_election.cc:288] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:09.501639 22434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:09.501816 22434 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [term 1 LEADER]: Becoming Leader. State: Replica: ee1ed9e54e7649db937b380a5d2459f0, State: Running, Role: LEADER
I20260927 21:02:09.501889 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:09.501932 22434 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } }
I20260927 21:02:09.502321 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee1ed9e54e7649db937b380a5d2459f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } } }
I20260927 21:02:09.502426 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:09.502635 22441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:09.502806 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee1ed9e54e7649db937b380a5d2459f0. Latest consensus state: current_term: 1 leader_uuid: "ee1ed9e54e7649db937b380a5d2459f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee1ed9e54e7649db937b380a5d2459f0" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43851 } } }
I20260927 21:02:09.503082 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:09.503134 22441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:09.505024 22441 catalog_manager.cc:1347] Generated new cluster ID: 02d9413334e04d01b778aff6234aaee7
I20260927 21:02:09.505081 22441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:09.510053 22441 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:09.510614 22441 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:09.514586 22441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee1ed9e54e7649db937b380a5d2459f0: Generated new TSK 0
I20260927 21:02:09.514775 22441 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 21:02:09.538661 22459 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 21:02:09.538762 22459 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 21:02:09.539084 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.524091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.16.190.254:43851)
0927 21:02:09.524157 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.524159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.524266 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:02:09.524346 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.524347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.524357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.524417 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:02:09.524578 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.525122 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.526287 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.526479 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.526527 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.526739 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:02:09.538893 (+ 12154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:02:09.539027 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 21:02:09.539173 22433 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 21:02:09.539294 22458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:09.524027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:43851 (local address 127.0.0.1:34168)
0927 21:02:09.524187 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.524198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:09.524273 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:09.524440 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:09.524441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:09.524444 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:02:09.524510 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.524512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.525214 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.525216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.526190 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.526193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.526552 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.526554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.526621 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.526627 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:02:09.539018 (+ 12391us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 21:02:09.539182 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.190.254:43851: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260927 21:02:09.539645 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22362
2026-09-27T21:02:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (274 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 21:02:09.577860 17147 mini_oidc.cc:152] Starting JWKS server
I20260927 21:02:09.578612 17147 webserver.cc:460] Webserver started at https://127.0.0.1:39021/ using document root <none> and password file <none>
I20260927 21:02:09.578685 17147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 21:02:09.578845 17147 webserver.cc:460] Webserver started at http://127.0.0.1:40645/ using document root <none> and password file <none>
2026-09-27T21:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:09Z Disabled control of system clock
I20260927 21:02:09.587881 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42933
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39021/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:02:09.670773 22465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:09.671038 22465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:09.671063 22465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:09.672586 22465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:09.672654 22465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:09.672667 22465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:09.672680 22465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:09.672694 22465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:02:09.672716 22465 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:02:09.672785 22465 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39021/jwks.json
I20260927 21:02:09.674551 22465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39021/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:09.674862 22465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:09.675189 22465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:09.680377 22471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.680372 22473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:09.680460 22465 server_base.cc:1034] running on GCE node
W20260927 21:02:09.680372 22470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:09.680855 22465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:09.681162 22465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:09.682315 22465 hybrid_clock.cc:648] HybridClock initialized: now 1790542929682304 us; error 31 us; skew 500 ppm
I20260927 21:02:09.683667 22465 webserver.cc:460] Webserver started at http://127.16.190.254:43427/ using document root <none> and password file <none>
I20260927 21:02:09.683926 22465 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:09.684005 22465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:09.684159 22465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:09.685380 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "595afeaf606c4d1597f5bd19fe8f1b77"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "08d24559152068c8002c46b23c4abd79"
server_key_iv: "35021ee50caa99308c9fb8af0c49a4cc"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.685768 22465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "595afeaf606c4d1597f5bd19fe8f1b77"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "08d24559152068c8002c46b23c4abd79"
server_key_iv: "35021ee50caa99308c9fb8af0c49a4cc"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.687139 22465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:09.687970 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:09.688126 22465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:09.688215 22465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "595afeaf606c4d1597f5bd19fe8f1b77"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "08d24559152068c8002c46b23c4abd79"
server_key_iv: "35021ee50caa99308c9fb8af0c49a4cc"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.688284 22465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:09.696158 22465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:09.696442 22465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:09.696555 22465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:09.700522 22465 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:42933
I20260927 21:02:09.700626 22531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:42933 every 8 connection(s)
I20260927 21:02:09.701017 22465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:09.701797 22532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:09.702469 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22465
I20260927 21:02:09.702550 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:09.702754 17147 external_mini_cluster.cc:1428] Setting key 22f86f733f0a42e22a066c9816609753
I20260927 21:02:09.706233 22532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: Bootstrap starting.
I20260927 21:02:09.706862 22532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:09.707173 22532 log.cc:824] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:09.708005 22532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: No bootstrap required, opened a new log
I20260927 21:02:09.709654 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.703329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.16.190.254:42933)
0927 21:02:09.703531 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.703540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.703574 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:02:09.704296 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.704306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.704394 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.704560 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:09.704805 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.706187 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.707387 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.707977 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.708177 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.708887 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:09.708891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:09.708892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:09.709104 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:09.709146 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:09.709162 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:09.709235 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:09.709340 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:02:09.709428 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260927 21:02:09.710759 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:02:09.710737 22532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } }
I20260927 21:02:09.710920 22532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:09.710961 22532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595afeaf606c4d1597f5bd19fe8f1b77, State: Initialized, Role: FOLLOWER
I20260927 21:02:09.711091 22532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } }
I20260927 21:02:09.711159 22532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:09.711181 22532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:09.711249 22532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:09.803107 22532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } }
I20260927 21:02:09.803361 22532 leader_election.cc:302] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 595afeaf606c4d1597f5bd19fe8f1b77; no voters:
I20260927 21:02:09.803654 22532 leader_election.cc:288] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:09.803900 22537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:09.804014 22532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:09.804332 22537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [term 1 LEADER]: Becoming Leader. State: Replica: 595afeaf606c4d1597f5bd19fe8f1b77, State: Running, Role: LEADER
I20260927 21:02:09.804654 22537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } }
I20260927 21:02:09.805428 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } } }
I20260927 21:02:09.805864 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:09.806576 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 595afeaf606c4d1597f5bd19fe8f1b77. Latest consensus state: current_term: 1 leader_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "595afeaf606c4d1597f5bd19fe8f1b77" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42933 } } }
I20260927 21:02:09.806650 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77 [sys.catalog]: This master's current role is: LEADER
W20260927 21:02:09.806253 22551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:02:09.806733 22551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:02:09.806910 22553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:09.807116 22553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:09.808647 22553 catalog_manager.cc:1347] Generated new cluster ID: 1bb3f64a6a1448f68a7871650992f84b
I20260927 21:02:09.808713 22553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:09.814363 22553 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:09.815096 22553 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:09.819185 22553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 595afeaf606c4d1597f5bd19fe8f1b77: Generated new TSK 0
I20260927 21:02:09.819391 22553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:09.833137 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:09.829065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.16.190.254:42933)
0927 21:02:09.829116 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:09.829119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:09.829284 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:02:09.829335 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:09.829336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:09.829350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:09.829433 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:02:09.829624 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.830181 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.831341 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.831535 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.831604 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.831847 (+ 243us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:02:09.832909 (+ 1062us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39021/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:02:09.833060 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39021/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 21:02:09.833199 22535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44724: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39021/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 21:02:09.833230 22561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:09.828969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42933 (local address 127.0.0.1:44724)
0927 21:02:09.829142 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:09.829154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:09.829229 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:09.829448 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:09.829449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:09.829451 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:02:09.829526 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.829529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.830277 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.830278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.831251 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:09.831254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:09.831634 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:09.831635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:09.831710 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:09.831716 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:02:09.833022 (+ 1306us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39021/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 21:02:09.833126 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.190.254:42933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39021/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260927 21:02:09.835104 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22465
E20260927 21:02:09.844116 22562 webserver.cc:533] Webserver: sslize failed:
E20260927 21:02:09.844210 22562 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T21:02:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (294 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T21:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:09Z Disabled control of system clock
I20260927 21:02:09.880383 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36193
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36193
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/testchainca.cert with env {}
W20260927 21:02:09.967374 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:09.967630 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:09.967656 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:09.968933 22566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:09.968976 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:09.968989 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:09.969002 22566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:09.969013 22566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 21:02:09.969025 22566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 21:02:09.969036 22566 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 21:02:09.970547 22566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:09.970754 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:09.970923 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:09.976347 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.976336 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:09.976336 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:09.976613 22566 server_base.cc:1034] running on GCE node
I20260927 21:02:09.977680 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:09.978045 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:09.979216 22566 hybrid_clock.cc:648] HybridClock initialized: now 1790542929979187 us; error 45 us; skew 500 ppm
I20260927 21:02:09.980933 22566 webserver.cc:460] Webserver started at http://127.16.190.254:34875/ using document root <none> and password file <none>
I20260927 21:02:09.981277 22566 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:09.981338 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:09.981537 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:09.982841 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "aa3851630f0c4ec2a215da2dd7e819fb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "7f42ac2e0053a0e3ba2e291c8d5df172"
server_key_iv: "3076b54a43b4716234f4eaf77f66d041"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.983270 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "aa3851630f0c4ec2a215da2dd7e819fb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "7f42ac2e0053a0e3ba2e291c8d5df172"
server_key_iv: "3076b54a43b4716234f4eaf77f66d041"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.984747 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260927 21:02:09.985725 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:09.985989 22566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:09.986074 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "aa3851630f0c4ec2a215da2dd7e819fb"
format_stamp: "Formatted at 2026-09-27 21:02:09 on dist-test-slave-6c7s"
server_key: "7f42ac2e0053a0e3ba2e291c8d5df172"
server_key_iv: "3076b54a43b4716234f4eaf77f66d041"
server_key_version: "encryptionkey@0"
I20260927 21:02:09.986181 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:10.010366 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:10.010749 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:10.010934 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:10.015801 22566 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36193
I20260927 21:02:10.015937 22632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36193 every 8 connection(s)
I20260927 21:02:10.016316 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:10.017407 22633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:10.019729 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb: Bootstrap starting.
I20260927 21:02:10.020330 22633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:10.020673 22633 log.cc:824] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:10.021591 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb: No bootstrap required, opened a new log
I20260927 21:02:10.024489 22633 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } }
I20260927 21:02:10.024694 22633 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:10.024741 22633 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3851630f0c4ec2a215da2dd7e819fb, State: Initialized, Role: FOLLOWER
I20260927 21:02:10.024936 22633 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } }
I20260927 21:02:10.025023 22633 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:10.025068 22633 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:10.025132 22633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:10.025480 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22566
I20260927 21:02:10.025601 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:10.025797 17147 external_mini_cluster.cc:1428] Setting key 556886042a798ac990040336a777db58
I20260927 21:02:10.031647 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:10.026421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.16.190.254:36193)
0927 21:02:10.026667 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:10.026678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:10.026716 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:02:10.026843 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:10.026852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:10.026928 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:10.027094 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:10.027451 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.029128 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.029885 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.030290 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.030378 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:10.031083 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:10.031086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:10.031087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:10.031250 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:10.031283 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:10.031293 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:10.031346 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:10.031410 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 21:02:10.031471 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260927 21:02:10.032584 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:02:10.103626 22633 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } }
I20260927 21:02:10.103844 22633 leader_election.cc:302] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa3851630f0c4ec2a215da2dd7e819fb; no voters:
I20260927 21:02:10.104158 22633 leader_election.cc:288] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:10.104312 22638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:10.104481 22638 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [term 1 LEADER]: Becoming Leader. State: Replica: aa3851630f0c4ec2a215da2dd7e819fb, State: Running, Role: LEADER
I20260927 21:02:10.104552 22633 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:10.104594 22638 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } }
I20260927 21:02:10.104986 22638 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3851630f0c4ec2a215da2dd7e819fb. Latest consensus state: current_term: 1 leader_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } } }
I20260927 21:02:10.105082 22638 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:10.104992 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa3851630f0c4ec2a215da2dd7e819fb" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36193 } } }
I20260927 21:02:10.105279 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:10.105322 22644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:10.105844 22644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:10.107909 22644 catalog_manager.cc:1347] Generated new cluster ID: 1556bb2db794403bae08a92a3a57f025
I20260927 21:02:10.107995 22644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:10.122160 22644 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:10.122696 22644 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:10.128629 22644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa3851630f0c4ec2a215da2dd7e819fb: Generated new TSK 0
I20260927 21:02:10.128872 22644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:10.286389 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:10.138458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.16.190.254:36193)
0927 21:02:10.138508 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:10.138510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:10.138592 (+ 82us) server_negotiation.cc:407] Connection header received
0927 21:02:10.138661 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:10.138662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:10.138675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:10.138752 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0927 21:02:10.138923 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.139713 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.140502 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.140840 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.140903 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:10.141094 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 21:02:10.286139 (+145045us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:02:10.286321 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47914: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 21:02:10.286458 22637 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47914: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 21:02:10.286715 22662 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:10.138368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36193 (local address 127.0.0.1:47914)
0927 21:02:10.138504 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:10.138525 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:10.138589 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:10.138783 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:10.138784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:10.138786 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 21:02:10.138843 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:10.138845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.139822 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.139823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:10.140413 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:10.140415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:10.140941 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:10.140942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:10.140985 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:10.140990 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 21:02:10.286315 (+145325us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 21:02:10.286442 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.190.254:36193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":92,"threads_started":1}
I20260927 21:02:10.287087 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22566
2026-09-27T21:02:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (432 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (92 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:12 dist-test-slave-6c7s krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:12 dist-test-slave-6c7s krb5kdc[22674](info): set up 2 sockets
Sep 27 21:02:12 dist-test-slave-6c7s krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260927 21:02:13.641836 17147 mini_kdc.cc:121] Time spent starting KDC: real 3.148s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:13 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:13.764750 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:43609
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:43609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:13.847622 22687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:13.847899 22687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:13.847939 22687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:13.849166 22687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:13.849251 22687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:13.849273 22687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:13.849295 22687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:13.849325 22687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:13.850847 22687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:43609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:43609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:13.851171 22687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:13.851418 22687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:13.856652 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:13.856581 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:13.856571 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:13.856849 22687 server_base.cc:1034] running on GCE node
I20260927 21:02:13.857167 22687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:13.857471 22687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:13.858654 22687 hybrid_clock.cc:648] HybridClock initialized: now 1790542933858643 us; error 35 us; skew 500 ppm
Sep 27 21:02:13 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542933, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:13.865628 22687 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:13.866122 22687 webserver.cc:460] Webserver started at http://127.16.190.254:35007/ using document root <none> and password file <none>
I20260927 21:02:13.866384 22687 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:13.866453 22687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:13.866607 22687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:13.867725 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "90c4c6bd09ec4a6e890d5a515de91f3b"
format_stamp: "Formatted at 2026-09-27 21:02:13 on dist-test-slave-6c7s"
server_key: "a2bd0856cd3fe5f3aa9ec776d00533ad"
server_key_iv: "9385a181430e1bc3981ee25827acfebf"
server_key_version: "encryptionkey@0"
I20260927 21:02:13.868108 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "90c4c6bd09ec4a6e890d5a515de91f3b"
format_stamp: "Formatted at 2026-09-27 21:02:13 on dist-test-slave-6c7s"
server_key: "a2bd0856cd3fe5f3aa9ec776d00533ad"
server_key_iv: "9385a181430e1bc3981ee25827acfebf"
server_key_version: "encryptionkey@0"
I20260927 21:02:13.869619 22687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:13.870438 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:13.870641 22687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:13.870755 22687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "90c4c6bd09ec4a6e890d5a515de91f3b"
format_stamp: "Formatted at 2026-09-27 21:02:13 on dist-test-slave-6c7s"
server_key: "a2bd0856cd3fe5f3aa9ec776d00533ad"
server_key_iv: "9385a181430e1bc3981ee25827acfebf"
server_key_version: "encryptionkey@0"
I20260927 21:02:13.870836 22687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:13.902010 22687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:13.904265 22687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:13.904440 22687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:13.908510 22687 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:43609
I20260927 21:02:13.908545 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:43609 every 8 connection(s)
I20260927 21:02:13.908999 22687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:13.909940 22755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:13.910033 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22687
I20260927 21:02:13.910157 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:13.910348 17147 external_mini_cluster.cc:1428] Setting key 8897227ce715cfd980b4ed5cfa2f1987
I20260927 21:02:13.913581 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b: Bootstrap starting.
I20260927 21:02:13.914271 22755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:13.914573 22755 log.cc:824] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:13.915516 22755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b: No bootstrap required, opened a new log
Sep 27 21:02:13 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:13.918138 22755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } }
I20260927 21:02:13.918304 22755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:13.918351 22755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c4c6bd09ec4a6e890d5a515de91f3b, State: Initialized, Role: FOLLOWER
I20260927 21:02:13.918490 22755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } }
I20260927 21:02:13.918586 22755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:13.918630 22755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:13.918673 22755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:13.919222 22755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } }
I20260927 21:02:13.919342 22755 leader_election.cc:302] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c4c6bd09ec4a6e890d5a515de91f3b; no voters:
I20260927 21:02:13.919535 22755 leader_election.cc:288] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:13.919595 22760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:13.919744 22760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [term 1 LEADER]: Becoming Leader. State: Replica: 90c4c6bd09ec4a6e890d5a515de91f3b, State: Running, Role: LEADER
I20260927 21:02:13.919797 22755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:13.919921 22760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } }
I20260927 21:02:13.920416 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } } }
I20260927 21:02:13.920650 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:13.920570 22762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c4c6bd09ec4a6e890d5a515de91f3b. Latest consensus state: current_term: 1 leader_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c4c6bd09ec4a6e890d5a515de91f3b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 43609 } } }
I20260927 21:02:13.921128 22762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:13.921254 22769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:13.921774 22769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:13.922379 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:13.910969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.16.190.254:43609)
0927 21:02:13.911171 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:13.911182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:13.911219 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:02:13.911722 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:13.911731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:13.911801 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:13.911965 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:13.912592 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:13.914008 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:13.914818 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:13.915389 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:13.915474 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:13.918212 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:13.918218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:13.918219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:13.918235 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:13.920287 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:13.920751 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:13.920754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:13.920754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:13.920878 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:13.921113 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:13.921114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:13.921115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:13.921454 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:13.921632 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:13.922048 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 21:02:13.922183 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260927 21:02:13.923401 22769 catalog_manager.cc:1347] Generated new cluster ID: 02e2c7a599134593bb714a6174de17ba
I20260927 21:02:13.923441 22769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:13.935067 22769 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:13.935667 22769 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:13.940313 22769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90c4c6bd09ec4a6e890d5a515de91f3b: Generated new TSK 0
I20260927 21:02:13.940521 22769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:13.975698 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:14.074470 22781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:14.074717 22781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:14.074744 22781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:14.076527 22781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:14.076577 22781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:14.076608 22781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:14.078222 22781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:14.078503 22781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:14.078771 22781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:14.084048 22787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:14.084092 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:14.084285 22786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:14.084187 22781 server_base.cc:1034] running on GCE node
I20260927 21:02:14.084510 22781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:14.084735 22781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:14.085903 22781 hybrid_clock.cc:648] HybridClock initialized: now 1790542934085879 us; error 38 us; skew 500 ppm
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.088575 22781 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:14.089130 22781 webserver.cc:460] Webserver started at http://127.16.190.193:33651/ using document root <none> and password file <none>
I20260927 21:02:14.089402 22781 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:14.089452 22781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:14.089622 22781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:14.090849 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "af56e063c99d4ef1bb16ed923874edf3"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "1d7bb8e4a328eb4fe43b67b0fb2b0f9e"
server_key_iv: "3a2b5de57086a99fb676d3c0efbfb846"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.091240 22781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "af56e063c99d4ef1bb16ed923874edf3"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "1d7bb8e4a328eb4fe43b67b0fb2b0f9e"
server_key_iv: "3a2b5de57086a99fb676d3c0efbfb846"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.092726 22781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:14.093690 22796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.093905 22781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:14.094048 22781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "af56e063c99d4ef1bb16ed923874edf3"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "1d7bb8e4a328eb4fe43b67b0fb2b0f9e"
server_key_iv: "3a2b5de57086a99fb676d3c0efbfb846"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.094127 22781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:14.111790 22781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:14.114367 22781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:14.114524 22781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:14.114794 22781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:14.115135 22781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:14.115191 22781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.115234 22781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:14.115262 22781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.121793 22781 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:39685
I20260927 21:02:14.121836 22909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:39685 every 8 connection(s)
I20260927 21:02:14.122193 22781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:14.131835 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22781
I20260927 21:02:14.131945 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:14.132102 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.123153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:55843 (local address 127.16.190.254:43609)
0927 21:02:14.123240 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.123242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.123416 (+ 174us) server_negotiation.cc:407] Connection header received
0927 21:02:14.124379 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.124380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.124391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.124457 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.125158 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.125674 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.126673 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.126874 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.126930 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.129202 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.129204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.129204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.129210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.130754 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.131142 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.131143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.131143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.131198 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.131419 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.131420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.131420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.131523 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.131574 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.131887 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.131997 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 21:02:14.132144 17147 external_mini_cluster.cc:1428] Setting key 375192ce8902c165ce114d9ad10125b4
I20260927 21:02:14.132264 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.123121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:43609 (local address 127.16.190.193:55843)
0927 21:02:14.123277 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:14.123321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:14.124263 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:14.124573 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:14.124577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:14.124853 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:14.125055 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.125059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.125764 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.125765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.126584 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.126587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.126952 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.126953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.127061 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.127551 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:14.127559 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:14.129101 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:14.130847 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.130853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.130855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.131068 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.131264 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.131264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.131265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.131363 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.131583 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:14.131588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:14.131751 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 21:02:14.131839 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:02:14.132020 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260927 21:02:14.132880 22910 heartbeater.cc:347] Connected to a master server at 127.16.190.254:43609
I20260927 21:02:14.133008 22910 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:14.133190 22910 heartbeater.cc:511] Master 127.16.190.254:43609 requested a full tablet report, sending...
I20260927 21:02:14.133808 22718 ts_manager.cc:194] Registered new tserver with Master: af56e063c99d4ef1bb16ed923874edf3 (127.16.190.193:39685)
I20260927 21:02:14.134622 22718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:55843
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:14.161821 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:14.246300 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:14.246578 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:14.246618 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:14.247786 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:14.247826 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:14.247853 22915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:14.249332 22915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:14.249538 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:14.249728 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:14.255071 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:14.255071 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:14.255126 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:14.255420 22915 server_base.cc:1034] running on GCE node
I20260927 21:02:14.255645 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:14.255867 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:14.257040 22915 hybrid_clock.cc:648] HybridClock initialized: now 1790542934257007 us; error 55 us; skew 500 ppm
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.260123 22915 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:14.260715 22915 webserver.cc:460] Webserver started at http://127.16.190.194:42231/ using document root <none> and password file <none>
I20260927 21:02:14.261044 22915 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:14.261117 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:14.261276 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:14.262396 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "85c927750d944dc5a3c5fe958ae962c7"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "2d0dc318e6f59bcb5415751aa88851aa"
server_key_iv: "037e70b67e260b5fca666ee89794afd8"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.262791 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "85c927750d944dc5a3c5fe958ae962c7"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "2d0dc318e6f59bcb5415751aa88851aa"
server_key_iv: "037e70b67e260b5fca666ee89794afd8"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.264236 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:14.265112 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.265292 22915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:14.265386 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "85c927750d944dc5a3c5fe958ae962c7"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "2d0dc318e6f59bcb5415751aa88851aa"
server_key_iv: "037e70b67e260b5fca666ee89794afd8"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.265472 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:14.279071 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:14.281620 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:14.281796 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:14.282054 22915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:14.282387 22915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:14.282444 22915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.282493 22915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:14.282542 22915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.288470 22915 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:36859
I20260927 21:02:14.288511 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:36859 every 8 connection(s)
I20260927 21:02:14.288877 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:14.297564 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 22915
I20260927 21:02:14.297680 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:14.297901 17147 external_mini_cluster.cc:1428] Setting key 0727e932ccdfb1e17e3f5f3082a27b80
I20260927 21:02:14.298982 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.289709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:52201 (local address 127.16.190.254:43609)
0927 21:02:14.289787 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.289789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.289977 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:02:14.290909 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.290910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.290919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.290983 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.291588 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.292083 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.293167 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.293374 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.293431 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.295744 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.295745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.295745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.295757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.297198 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.297826 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.297827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.297828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.297882 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.298238 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.298239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.298239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.298373 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.298427 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.298782 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.298870 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:02:14.299263 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.289684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:43609 (local address 127.16.190.194:52201)
0927 21:02:14.289840 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:14.289882 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:14.290812 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:14.291044 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:14.291047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:14.291309 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:14.291512 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.291516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.292175 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.292176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.293090 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.293092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.293472 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.293473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.293611 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.294177 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:14.294186 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:14.295615 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:14.297321 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.297328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.297330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.297689 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.297955 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.297956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.297956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.298142 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.298439 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:14.298445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:14.298612 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:02:14.298727 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:02:14.298932 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260927 21:02:14.299782 23044 heartbeater.cc:347] Connected to a master server at 127.16.190.254:43609
I20260927 21:02:14.299892 23044 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:14.300084 23044 heartbeater.cc:511] Master 127.16.190.254:43609 requested a full tablet report, sending...
I20260927 21:02:14.300487 22718 ts_manager.cc:194] Registered new tserver with Master: 85c927750d944dc5a3c5fe958ae962c7 (127.16.190.194:36859)
I20260927 21:02:14.300962 22718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:52201
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:14.324749 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:14.406818 23049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:14.407033 23049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:14.407056 23049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:14.408154 23049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:14.408186 23049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:14.408212 23049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:14.409631 23049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:43609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:14.409829 23049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:14.410005 23049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:14.415119 23055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:14.415230 23057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:14.415390 23049 server_base.cc:1034] running on GCE node
W20260927 21:02:14.415496 23054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:14.415746 23049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:14.415961 23049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:14.417110 23049 hybrid_clock.cc:648] HybridClock initialized: now 1790542934417086 us; error 39 us; skew 500 ppm
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.419940 23049 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:14.420475 23049 webserver.cc:460] Webserver started at http://127.16.190.195:33347/ using document root <none> and password file <none>
I20260927 21:02:14.420711 23049 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:14.420758 23049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:14.420946 23049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:14.422395 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "1cf1662418a545bf83574a95d4fee429"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "91bdd34920648190e5ab99f077d3461d"
server_key_iv: "eb85dd67e34a4560707782b94302fe81"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.422852 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "1cf1662418a545bf83574a95d4fee429"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "91bdd34920648190e5ab99f077d3461d"
server_key_iv: "eb85dd67e34a4560707782b94302fe81"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.424276 23049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:14.425170 23064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.425429 23049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:14.425530 23049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "1cf1662418a545bf83574a95d4fee429"
format_stamp: "Formatted at 2026-09-27 21:02:14 on dist-test-slave-6c7s"
server_key: "91bdd34920648190e5ab99f077d3461d"
server_key_iv: "eb85dd67e34a4560707782b94302fe81"
server_key_version: "encryptionkey@0"
I20260927 21:02:14.425649 23049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:14.438381 23049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:14.440971 23049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:14.441138 23049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:14.441377 23049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:14.441702 23049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:14.441736 23049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.441759 23049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:14.441771 23049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:14.447634 23049 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:40517
I20260927 21:02:14.447706 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:40517 every 8 connection(s)
I20260927 21:02:14.448027 23049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:14.449817 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23049
I20260927 21:02:14.449919 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:14.450078 17147 external_mini_cluster.cc:1428] Setting key bb97f9630a4eabbacf81b3da5df96c37
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:14.458437 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.448842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:42013 (local address 127.16.190.254:43609)
0927 21:02:14.448911 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.448913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.449101 (+ 188us) server_negotiation.cc:407] Connection header received
0927 21:02:14.450244 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.450246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.450260 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.450332 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.451035 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.451558 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.452612 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.452901 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.452991 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.455239 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.455240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.455240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.455249 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.456649 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.457338 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.457340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.457340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.457411 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.457745 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.457746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.457746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.457873 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.457950 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.458251 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.458336 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:02:14.458681 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.448746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:43609 (local address 127.16.190.195:42013)
0927 21:02:14.448956 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:14.449002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:14.450091 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:14.450409 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:14.450413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:14.450715 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:14.450918 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.450923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.451659 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.451660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.452506 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:14.452508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.453002 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.453003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:14.453140 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.453703 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:14.453711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:14.455127 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:14.456805 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.456957 (+ 152us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.456960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.457216 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.457497 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:14.457498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:14.457499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:14.457667 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:14.457950 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:14.457958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:14.458113 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:02:14.458229 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:02:14.458405 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260927 21:02:14.459407 23178 heartbeater.cc:347] Connected to a master server at 127.16.190.254:43609
I20260927 21:02:14.459501 23178 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:14.459775 23178 heartbeater.cc:511] Master 127.16.190.254:43609 requested a full tablet report, sending...
I20260927 21:02:14.460256 22718 ts_manager.cc:194] Registered new tserver with Master: 1cf1662418a545bf83574a95d4fee429 (127.16.190.195:40517)
I20260927 21:02:14.460673 22718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:42013
I20260927 21:02:14.461866 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/test-admin.keytab.
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.477922 17147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 21:02:14.479791 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:14.478819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.16.190.193:39685)
0927 21:02:14.479024 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.479030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.479053 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:02:14.479088 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.479090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.479107 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.479189 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.479641 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47132: BlockingRecv error: recv got EOF from 127.0.0.1:47132 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
W20260927 21:02:14.479766 17147 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc)
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.481190 17147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.482319 17147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:02:14.490227 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.482909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.16.190.193:39685)
0927 21:02:14.482946 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.482948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.483005 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:02:14.483056 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.483057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.483066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.483122 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.483485 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.484246 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.485054 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.485381 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.485449 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.487585 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.487592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.487593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.487600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.489088 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.489413 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.489416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.489416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.489470 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.489641 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.489641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.489641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.489782 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.489893 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.490085 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.490149 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 21:02:14.492286 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:14.491580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.16.190.193:39685)
0927 21:02:14.491636 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.491638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.491697 (+ 59us) server_negotiation.cc:407] Connection header received
0927 21:02:14.491770 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.491771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.491781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.491835 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.492237 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47158: BlockingRecv error: recv got EOF from 127.0.0.1:47158 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 21:02:14.492524 17147 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.493611 17147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.494611 17147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:02:14.502782 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.495195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.16.190.193:39685)
0927 21:02:14.495255 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.495257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.495447 (+ 190us) server_negotiation.cc:407] Connection header received
0927 21:02:14.495472 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.495473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.495487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.495539 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.496075 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.496773 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.497907 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.498109 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.498168 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.500320 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.500322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.500322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.500329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.501858 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.502148 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.502149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.502149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.502192 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.502338 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.502339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.502339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.502419 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.502486 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.502568 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.502627 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 21:02:14.504314 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:14.503832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.16.190.193:39685)
0927 21:02:14.503869 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.503871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.503901 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:02:14.503950 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.503951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.503958 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.504003 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.504280 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47180: BlockingRecv error: recv got EOF from 127.0.0.1:47180 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:02:14.504405 17147 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790542906842326-17147-0/krb5kdc/krb5cc' of type FILE
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.505438 17147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:14.506290 17147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:02:14.513545 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:14.506829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.16.190.193:39685)
0927 21:02:14.506889 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:14.506892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:14.506938 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:02:14.507014 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:14.507015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:14.507024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:14.507093 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:14.507499 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.507978 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.508686 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:14.508946 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:14.509023 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:14.511058 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:14.511059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:14.511059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:14.511066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:14.512256 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.512635 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.512635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.512636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.512684 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:14.513045 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:14.513046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:14.513046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:14.513159 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:14.513264 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:14.513397 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:02:14.513457 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:02:14.514333 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22781
I20260927 21:02:14.519539 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22915
I20260927 21:02:14.524530 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23049
I20260927 21:02:14.528607 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 22687
2026-09-27T21:02:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4055 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[23198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[23198](info): set up 2 sockets
Sep 27 21:02:14 dist-test-slave-6c7s krb5kdc[23198](info): commencing operation
krb5kdc: starting...
W20260927 21:02:15.573823 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:15 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab.
Entry for principal oryx/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab.
I20260927 21:02:15.696434 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37891
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab
--principal=oryx/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:15.776305 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:15.776525 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:15.776546 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:15.777673 23211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:15.777720 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:15.777734 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:15.777746 23211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:15.777758 23211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:15.779153 23211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab
--principal=oryx/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:15.779361 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:15.779538 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:15.784364 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:15.784358 23217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:15.784358 23216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:15.784446 23211 server_base.cc:1034] running on GCE node
I20260927 21:02:15.784940 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:15.785259 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:15.786424 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790542935786414 us; error 31 us; skew 500 ppm
Sep 27 21:02:15 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:15.788878 23211 init.cc:377] Logged in from keytab as oryx/127.16.190.254@KRBTEST.COM (short username oryx)
I20260927 21:02:15.789352 23211 webserver.cc:460] Webserver started at http://127.16.190.254:42033/ using document root <none> and password file <none>
I20260927 21:02:15.789588 23211 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:15.789657 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:15.789819 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:15.790916 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "66b0334abc594ee6832dd56fe5e8d268"
format_stamp: "Formatted at 2026-09-27 21:02:15 on dist-test-slave-6c7s"
server_key: "147e49608de5b5a286681a410bd284b3"
server_key_iv: "a4e5a953780fa02bfb5cfab92d2d1a19"
server_key_version: "encryptionkey@0"
I20260927 21:02:15.791291 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "66b0334abc594ee6832dd56fe5e8d268"
format_stamp: "Formatted at 2026-09-27 21:02:15 on dist-test-slave-6c7s"
server_key: "147e49608de5b5a286681a410bd284b3"
server_key_iv: "a4e5a953780fa02bfb5cfab92d2d1a19"
server_key_version: "encryptionkey@0"
I20260927 21:02:15.792603 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:15.793434 23226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:15.793618 23211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:15.793718 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "66b0334abc594ee6832dd56fe5e8d268"
format_stamp: "Formatted at 2026-09-27 21:02:15 on dist-test-slave-6c7s"
server_key: "147e49608de5b5a286681a410bd284b3"
server_key_iv: "a4e5a953780fa02bfb5cfab92d2d1a19"
server_key_version: "encryptionkey@0"
I20260927 21:02:15.793805 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:15.808593 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:15.810673 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:15.810868 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:15.815109 23211 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:37891
I20260927 21:02:15.815141 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:37891 every 8 connection(s)
I20260927 21:02:15.815500 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:15.816656 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:15.819198 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Bootstrap starting.
I20260927 21:02:15.819784 23279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:15.820135 23279 log.cc:824] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:15.820899 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: No bootstrap required, opened a new log
I20260927 21:02:15.821364 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23211
I20260927 21:02:15.821481 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:15.821677 17147 external_mini_cluster.cc:1428] Setting key 3e54634aa7cf9f88ac42306b21f8ae99
I20260927 21:02:15.823550 23279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } }
I20260927 21:02:15.823693 23279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:15.823715 23279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b0334abc594ee6832dd56fe5e8d268, State: Initialized, Role: FOLLOWER
I20260927 21:02:15.823850 23279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } }
I20260927 21:02:15.823925 23279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:15.823946 23279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:15.824011 23279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:15.824564 23279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } }
I20260927 21:02:15.824647 23279 leader_election.cc:302] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b0334abc594ee6832dd56fe5e8d268; no voters:
I20260927 21:02:15.824847 23279 leader_election.cc:288] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:15.824865 23284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:15.824977 23284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 LEADER]: Becoming Leader. State: Replica: 66b0334abc594ee6832dd56fe5e8d268, State: Running, Role: LEADER
I20260927 21:02:15.825066 23284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } }
I20260927 21:02:15.825107 23279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:15.825371 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } }
I20260927 21:02:15.825531 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:15.825727 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b0334abc594ee6832dd56fe5e8d268. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } }
I20260927 21:02:15.825853 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:15.826022 23290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:15.826526 23290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:15.828191 23290 catalog_manager.cc:1347] Generated new cluster ID: e5d5347804c04598b3ce8fd71157fd33
I20260927 21:02:15.828243 23290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:02:15 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:15.833815 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:15.822148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.16.190.254:37891)
0927 21:02:15.822339 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:15.822349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:15.822387 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:02:15.822806 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:15.822814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:15.822860 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:15.823007 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:15.823632 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:15.824758 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:15.825790 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:15.826417 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:15.826520 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:15.829485 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:15.829490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:15.829490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:15.829502 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:15.831678 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:15.832099 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:15.832102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:15.832103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:15.832206 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:15.832414 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:15.832414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:15.832415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:15.832730 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:15.832925 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:15.833454 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 21:02:15.833594 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260927 21:02:15.837252 23290 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:15.837805 23290 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:15.848670 23290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Generated new TSK 0
I20260927 21:02:15.848902 23290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
I20260927 21:02:15.884675 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:15.986438 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:15.986670 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:15.986696 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:15.987949 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:15.988013 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:15.988042 23305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:15.989732 23305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:15.990024 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:15.990285 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:15.995692 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:15.995771 23305 server_base.cc:1034] running on GCE node
W20260927 21:02:15.995704 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:15.995715 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:15.996100 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:15.996371 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:15.997628 23305 hybrid_clock.cc:648] HybridClock initialized: now 1790542935997600 us; error 39 us; skew 500 ppm
Sep 27 21:02:15 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.000514 23305 init.cc:377] Logged in from keytab as oryx/127.16.190.193@KRBTEST.COM (short username oryx)
I20260927 21:02:16.001098 23305 webserver.cc:460] Webserver started at http://127.16.190.193:41377/ using document root <none> and password file <none>
I20260927 21:02:16.001330 23305 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.001374 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.001473 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:16.002532 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "9f98770f6825405a98b88d0780517753"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "e3c4702581ca33ea4d0cb178569188d6"
server_key_iv: "a468481555ff484f0876cd8b71e07162"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.002859 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "9f98770f6825405a98b88d0780517753"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "e3c4702581ca33ea4d0cb178569188d6"
server_key_iv: "a468481555ff484f0876cd8b71e07162"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.004254 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:16.005174 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.005386 23305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:16.005517 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "9f98770f6825405a98b88d0780517753"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "e3c4702581ca33ea4d0cb178569188d6"
server_key_iv: "a468481555ff484f0876cd8b71e07162"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.005616 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.014429 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.016788 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:16.017036 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.017302 23305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:16.017660 23305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:16.017725 23305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.017793 23305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:16.017823 23305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.025079 23305 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:43739
I20260927 21:02:16.025194 23433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:43739 every 8 connection(s)
I20260927 21:02:16.025601 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:16.030426 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23305
I20260927 21:02:16.030552 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:16.030819 17147 external_mini_cluster.cc:1428] Setting key c9ee5a0fabe019c067269b527cbba2fc
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:16.037983 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.026930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:52283 (local address 127.16.190.254:37891)
0927 21:02:16.026978 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.026980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.026991 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:02:16.027731 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.027733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.027747 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.027833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.029410 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.030199 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.031883 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.032100 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.032168 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.034922 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.034924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.034924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.034932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.036444 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.036922 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.036924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.036924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.036989 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.037257 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.037258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.037259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.037382 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.037464 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.037775 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.037884 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:16.038169 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.026573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:52283)
0927 21:02:16.026758 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.026811 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.027577 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.027980 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.027985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.028880 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.029232 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.029240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.030341 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.030343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.031791 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.031794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.032204 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.032205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.032377 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.033131 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.033142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.034806 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.036555 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.036562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.036565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.036857 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.037060 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.037061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.037061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.037182 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.037454 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.037460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.037624 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.037744 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.037915 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260927 21:02:16.038759 23434 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:16.038849 23434 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:16.039085 23434 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:16.039753 23243 ts_manager.cc:194] Registered new tserver with Master: 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739)
I20260927 21:02:16.040441 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:52283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
I20260927 21:02:16.060670 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:16.143268 23439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:16.143477 23439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:16.143501 23439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:16.144564 23439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:16.144599 23439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:16.144625 23439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:16.146090 23439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:16.146281 23439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:16.146451 23439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:16.151408 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.151454 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.151612 23439 server_base.cc:1034] running on GCE node
W20260927 21:02:16.151441 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.151880 23439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.152140 23439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.153327 23439 hybrid_clock.cc:648] HybridClock initialized: now 1790542936153321 us; error 39 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.155926 23439 init.cc:377] Logged in from keytab as oryx/127.16.190.194@KRBTEST.COM (short username oryx)
I20260927 21:02:16.156327 23439 webserver.cc:460] Webserver started at http://127.16.190.194:37687/ using document root <none> and password file <none>
I20260927 21:02:16.156520 23439 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.156560 23439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.156661 23439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:16.157972 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "693503666d254f348932e12a588a1448"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "434b505e988fa3c6a9ad2b10c8b27c7d"
server_key_iv: "0b884ec972ffe6be8b6a85bb34f21e44"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.158325 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "693503666d254f348932e12a588a1448"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "434b505e988fa3c6a9ad2b10c8b27c7d"
server_key_iv: "0b884ec972ffe6be8b6a85bb34f21e44"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.159727 23439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:16.160486 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.160706 23439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:16.160806 23439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "693503666d254f348932e12a588a1448"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "434b505e988fa3c6a9ad2b10c8b27c7d"
server_key_iv: "0b884ec972ffe6be8b6a85bb34f21e44"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.160935 23439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.183218 23439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.185798 23439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:16.185922 23439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.186126 23439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:16.186400 23439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:16.186439 23439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.186462 23439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:16.186474 23439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.192576 23439 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:41531
I20260927 21:02:16.192675 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:41531 every 8 connection(s)
I20260927 21:02:16.193053 23439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:16.195787 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23439
I20260927 21:02:16.195894 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:16.196087 17147 external_mini_cluster.cc:1428] Setting key 69617a74b2a589ec8387013ae2985657
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:16.204144 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.193896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:40309 (local address 127.16.190.254:37891)
0927 21:02:16.193985 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.193988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.194072 (+ 84us) server_negotiation.cc:407] Connection header received
0927 21:02:16.194660 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.194661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.194671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.194727 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.195743 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.196515 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.197792 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.198027 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.198105 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.200786 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.200788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.200788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.200797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.202627 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.203088 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.203089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.203090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.203156 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.203422 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.203423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.203423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.203586 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.203666 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.203930 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.204028 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:02:16.204239 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.193794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:40309)
0927 21:02:16.193944 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.194002 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.194516 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.194813 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.194816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.195409 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.195612 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.195617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.196629 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.196630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.197662 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.197665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.198136 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.198137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.198318 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.198920 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.198929 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.200667 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.202732 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.202741 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.202743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.202990 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.203245 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.203245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.203245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.203351 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.203656 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.203660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.203790 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.203878 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.204008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for oryx/127.16.190.195@KRBTEST.COM; defaulting to no policy
I20260927 21:02:16.204777 23568 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:16.204913 23568 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:16.205106 23568 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:16.205493 23243 ts_manager.cc:194] Registered new tserver with Master: 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:16.205905 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:40309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
I20260927 21:02:16.224920 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:16.307437 23573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:16.307655 23573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:16.307678 23573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:16.308964 23573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:16.309043 23573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:16.309093 23573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:16.310649 23573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:16.310849 23573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:16.311028 23573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:16.316167 23578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.316160 23581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.316160 23579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.316195 23573 server_base.cc:1034] running on GCE node
I20260927 21:02:16.316588 23573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.316859 23573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.318014 23573 hybrid_clock.cc:648] HybridClock initialized: now 1790542936318000 us; error 30 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.320562 23573 init.cc:377] Logged in from keytab as oryx/127.16.190.195@KRBTEST.COM (short username oryx)
I20260927 21:02:16.321125 23573 webserver.cc:460] Webserver started at http://127.16.190.195:35373/ using document root <none> and password file <none>
I20260927 21:02:16.321422 23573 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.321471 23573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.321623 23573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:16.322732 23573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "3e2213654f514bfbb47eb3b9b0ff9efa"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "a5a86b245996adc3278c69bb43e94a28"
server_key_iv: "9b833fcf82bb5c9294ebcfa9c0f3eb78"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.323104 23573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "3e2213654f514bfbb47eb3b9b0ff9efa"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "a5a86b245996adc3278c69bb43e94a28"
server_key_iv: "9b833fcf82bb5c9294ebcfa9c0f3eb78"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.324513 23573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:16.325359 23588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.325542 23573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.325639 23573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3e2213654f514bfbb47eb3b9b0ff9efa"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "a5a86b245996adc3278c69bb43e94a28"
server_key_iv: "9b833fcf82bb5c9294ebcfa9c0f3eb78"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.325717 23573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.335393 23573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.337337 23573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:16.337502 23573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.337726 23573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:16.338025 23573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:16.338057 23573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.338099 23573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:16.338125 23573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.344101 23573 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:35689
I20260927 21:02:16.344142 23701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:35689 every 8 connection(s)
I20260927 21:02:16.344475 23573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:16.349942 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23573
I20260927 21:02:16.350054 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:16.350260 17147 external_mini_cluster.cc:1428] Setting key 8f82410e73bc87e90da6439169c36002
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:16.353672 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.345251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:55325 (local address 127.16.190.254:37891)
0927 21:02:16.345331 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.345334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.345486 (+ 152us) server_negotiation.cc:407] Connection header received
0927 21:02:16.345956 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.345957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.345967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.346019 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.346904 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.347398 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.348493 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.348696 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.348788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.351214 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.351215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.351215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.351239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.352427 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.352799 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.352800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.352800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.352902 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.353121 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.353122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.353122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.353240 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.353293 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.353546 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.353607 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:02:16.353931 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.345184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:55325)
0927 21:02:16.345358 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.345394 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.345873 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.346076 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.346079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.346631 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.346830 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.346834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.347480 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.347481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.348422 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.348425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.348789 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.348790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.348966 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.349522 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.349530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.351101 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.352497 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.352503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.352505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.352731 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.352960 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.352961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.352961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.353063 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.353299 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.353304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.353433 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.353522 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.353645 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":114,"threads_started":1}
I20260927 21:02:16.354434 23702 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:16.354521 23702 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:16.354671 23702 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:16.355041 23243 ts_manager.cc:194] Registered new tserver with Master: 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:16.355378 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:55325
I20260927 21:02:16.362193 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.190.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.253.keytab.
Entry for principal oryx/127.16.190.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.253.keytab.
I20260927 21:02:16.388737 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:34651
--webserver_interface=127.16.190.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.253.keytab
--principal=oryx/127.16.190.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37891,127.16.190.253:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:16.471866 23707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:16.472146 23707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:16.472196 23707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:16.473445 23707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:16.473531 23707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:16.473554 23707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:16.473577 23707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:16.473606 23707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:16.475150 23707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37891,127.16.190.253:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.253.keytab
--principal=oryx/127.16.190.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.190.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
W20260927 21:02:16.475323 23707 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.190.254:37891,127.16.190.253:34651'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 21:02:16.475495 23707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:16.475737 23707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:16.480716 23713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.480691 23715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.480691 23712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.480969 23707 server_base.cc:1034] running on GCE node
I20260927 21:02:16.481324 23707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.481629 23707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.482800 23707 hybrid_clock.cc:648] HybridClock initialized: now 1790542936482801 us; error 41 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.485467 23707 init.cc:377] Logged in from keytab as oryx/127.16.190.253@KRBTEST.COM (short username oryx)
I20260927 21:02:16.485965 23707 webserver.cc:460] Webserver started at http://127.16.190.253:35051/ using document root <none> and password file <none>
I20260927 21:02:16.486220 23707 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.486289 23707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.486438 23707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:16.487802 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/instance:
uuid: "e50e1690dddf4d568459a7c3aa33abb6"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "d01b552d220ff7608444c5541c1344ef"
server_key_iv: "404eb8886aa1c40586912d5fda4274ce"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.488189 23707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal/instance:
uuid: "e50e1690dddf4d568459a7c3aa33abb6"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "d01b552d220ff7608444c5541c1344ef"
server_key_iv: "404eb8886aa1c40586912d5fda4274ce"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.490207 23707 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 21:02:16.491092 23722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.491292 23707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:16.491362 23707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "e50e1690dddf4d568459a7c3aa33abb6"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "d01b552d220ff7608444c5541c1344ef"
server_key_iv: "404eb8886aa1c40586912d5fda4274ce"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.491453 23707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.498603 23707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.500692 23707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:16.500886 23707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.504597 23707 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.253:34651
I20260927 21:02:16.504655 23774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.253:34651 every 8 connection(s)
I20260927 21:02:16.505045 23707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
I20260927 21:02:16.506210 23775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:16.507861 23775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:02:16.513934 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23707
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.253@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:16.518442 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.508799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.16.190.254:37891)
0927 21:02:16.508898 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.508900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.509192 (+ 292us) server_negotiation.cc:407] Connection header received
0927 21:02:16.509719 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.509720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.509731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.509847 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.510624 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.511127 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.512163 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.512359 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.512435 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.515136 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.515138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.515138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.515146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.516724 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.517216 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.517217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.517218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.517287 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.517677 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.517677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.517678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.517842 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.517936 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.518246 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.518347 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 21:02:16.518628 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.508813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:50956)
0927 21:02:16.509050 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.509104 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.509574 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.509962 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.509966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.510270 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.510490 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.510494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.511226 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.511227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.512065 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.512066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.512454 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.512455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.512594 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.513153 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.513162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.515041 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.516867 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.516873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.516876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.517134 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.517380 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.517381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.517381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.517592 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.517947 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.517952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.518092 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.518204 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.518367 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
I20260927 21:02:16.519260 23775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.253@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
I20260927 21:02:16.524286 23779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.514482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.16.190.253:34651)
0927 21:02:16.514756 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.514761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.514785 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:02:16.514809 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.514811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.514832 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.514918 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.515499 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.516398 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.517379 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.517824 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.517882 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.520935 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.520943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.520943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.520950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.522807 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.523191 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.523195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.523195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.523265 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.523483 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.523484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.523484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.523669 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.523861 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.523904 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.523983 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260927 21:02:16.526492 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.519579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36804)
0927 21:02:16.519632 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.519643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.519705 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.519942 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.519944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.520052 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.520102 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.520104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.520936 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.520937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.521472 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.521474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.521924 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.521925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.521989 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.522321 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.522325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.523596 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.525438 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.525439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.525439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.525600 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.525820 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.525820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.525820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.525872 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.526168 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.526170 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.526213 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.526282 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.526380 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:02:16.526573 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.519649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.16.190.253:34651)
0927 21:02:16.519812 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.519814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.519820 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:02:16.519839 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.519839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.519850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.519920 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.520185 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.520783 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.521579 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.521835 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.521925 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.523707 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.523708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.523708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.523714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.525347 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.525684 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.525685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.525685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.525738 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.525993 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.525994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.525994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.526095 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.526165 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.526307 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.526431 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260927 21:02:16.527796 23775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Bootstrap starting.
I20260927 21:02:16.528410 23775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:16.528736 23775 log.cc:824] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:16.529567 23775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: No bootstrap required, opened a new log
I20260927 21:02:16.530975 23775 raft_consensus.cc:348] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } }
I20260927 21:02:16.531100 23775 raft_consensus.cc:374] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:16.531123 23775 raft_consensus.cc:729] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50e1690dddf4d568459a7c3aa33abb6, State: Initialized, Role: FOLLOWER
I20260927 21:02:16.531216 23775 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } }
I20260927 21:02:16.531471 23775 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:16.531424 23782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } } }
I20260927 21:02:16.531545 23782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:02:16.533145 23793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:02:16.533227 23793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:02:16.534834 23707 master_runner.cc:428] Detected that this master e50e1690dddf4d568459a7c3aa33abb6 is joining an existing cluster
I20260927 21:02:16.534874 23707 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.190.253:34651
I20260927 21:02:16.542481 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.535914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.16.190.254:37891)
0927 21:02:16.535965 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.535968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.536099 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:02:16.536173 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.536174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.536182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.536247 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.536615 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.537313 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.538101 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.538301 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.538378 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.539400 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.539401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.539401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.539408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.541194 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.541606 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.541607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.541607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.541667 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.541888 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.541889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.541889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.542030 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.542105 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.542305 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.542389 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:02:16.542506 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.535853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:50962)
0927 21:02:16.536037 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.536048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.536112 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.536256 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.536257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.536463 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.536538 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.536540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.537397 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.537398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.537995 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.537997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.538376 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.538377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.538439 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.538939 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.538945 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.539287 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.541291 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.541292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.541292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.541511 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.541731 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.541732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.541732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.541809 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.542112 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.542113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.542163 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.542266 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.542367 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260927 21:02:16.543843 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.535895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36812)
0927 21:02:16.536912 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.536922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.536982 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.537131 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.537132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.537330 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.537390 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.537392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.538312 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.538313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.539120 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.539123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.539642 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.539643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.539708 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.540102 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.540107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.540489 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.542395 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.542395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.542396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.542591 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.542779 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.542780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.542781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.542833 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.543467 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.543468 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.543514 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.543628 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.543735 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":72,"threads_started":1}
I20260927 21:02:16.543915 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.536040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.16.190.253:34651)
0927 21:02:16.536080 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.536082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.536985 (+ 903us) server_negotiation.cc:407] Connection header received
0927 21:02:16.537032 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.537033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.537056 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.537117 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.537494 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.538222 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.539228 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.539531 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.539571 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.540584 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.540586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.540586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.540594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.542319 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.542656 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.542657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.542658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.542704 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.542906 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.542907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.542908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.543382 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.543453 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.543659 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.543801 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":23}
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.254@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
I20260927 21:02:16.552517 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.545483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.16.190.253:34651)
0927 21:02:16.545535 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.545537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.545703 (+ 166us) server_negotiation.cc:407] Connection header received
0927 21:02:16.545809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.545810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.545821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.545869 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.546295 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.546752 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.547531 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.547731 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.547804 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.549843 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.549844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.549845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.549852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.551254 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.551653 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.551655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.551655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.551710 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.551922 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.551923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.551923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.552041 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.552116 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.552316 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.552412 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:02:16.552529 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.545423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36818)
0927 21:02:16.545589 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.545635 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.545733 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.545872 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.545875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.546156 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.546204 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.546206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.546840 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.546841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.547434 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.547436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.547803 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.547804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.547906 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.548386 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.548390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.549746 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.551362 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.551369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.551372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.551575 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.551786 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.551787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.551787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.551847 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.552110 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.552117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.552169 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.552274 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.552414 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260927 21:02:16.553277 23243 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.190.253:34651: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6"
member_type: NON_VOTER
last_known_addr {
host: "127.16.190.253"
port: 34651
}
attrs {
promote: true
}
}
dest_uuid: "66b0334abc594ee6832dd56fe5e8d268"
cas_config_opid_index: -1
I20260927 21:02:16.553617 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } }
I20260927 21:02:16.554118 23803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } }
I20260927 21:02:16.554425 23749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:16.555272 23803 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.190.253:34651
I20260927 21:02:16.555320 23749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 21:02:16.555362 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } }
I20260927 21:02:16.555424 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:16.555619 23707 master.cc:537] Master@127.16.190.253:34651 shutting down...
I20260927 21:02:16.556133 23802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:02:16.557742 23227 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 21:02:16.558174 23227 consensus_peers.cc:487] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 -> Peer e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253:34651): Couldn't send request to peer e50e1690dddf4d568459a7c3aa33abb6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:02:16.558589 23707 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:16.558679 23707 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:16.558717 23707 tablet_replica.cc:330] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: stopping tablet replica
I20260927 21:02:16.572264 23707 master.cc:552] Master@127.16.190.253:34651 shutdown complete.
I20260927 21:02:16.572340 23707 master_runner.cc:305] Clearing existing system tablet
I20260927 21:02:16.572479 23707 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.572532 23707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.573356 23707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:02:16.573953 23807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.574124 23707 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.574189 23707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:16.574215 23707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "e50e1690dddf4d568459a7c3aa33abb6"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "d01b552d220ff7608444c5541c1344ef"
server_key_iv: "404eb8886aa1c40586912d5fda4274ce"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.575047 23707 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:16.576862 23707 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:02:16.576946 23707 log.cc:1196] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 21:02:16.577172 23707 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Deleting consensus metadata
I20260927 21:02:16.577298 23707 master_runner.cc:315] Copying system tablet from 127.16.190.254:37891
I20260927 21:02:16.577945 23707 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: tablet copy: Beginning tablet copy session from remote peer at address 127.16.190.254:37891
I20260927 21:02:16.584306 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.578274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:50978)
0927 21:02:16.578396 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.578409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.578509 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.578672 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.578673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.578873 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.578931 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.578932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.579636 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.579637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.580191 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.580193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.580595 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.580596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.580672 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.581222 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.581228 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.581534 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.583205 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.583206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.583207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.583392 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.583604 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.583605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.583605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.583668 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.583935 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.583935 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.583983 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.584073 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.584183 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260927 21:02:16.584292 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.578352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.16.190.254:37891)
0927 21:02:16.578409 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.578412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.578490 (+ 78us) server_negotiation.cc:407] Connection header received
0927 21:02:16.578594 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.578595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.578606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.578666 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.579028 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.579542 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.580292 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.580500 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.580593 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.581656 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.581658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.581658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.581666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.583098 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.583469 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.583470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.583470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.583530 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.583748 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.583749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.583750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.583866 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.583946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.584101 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.584182 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 21:02:16.584942 23263 tablet_copy_service.cc:140] P 66b0334abc594ee6832dd56fe5e8d268: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e50e1690dddf4d568459a7c3aa33abb6 ({username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:50978)
I20260927 21:02:16.585060 23263 tablet_copy_service.cc:161] P 66b0334abc594ee6832dd56fe5e8d268: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e50e1690dddf4d568459a7c3aa33abb6 at {username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:50978: session id = e50e1690dddf4d568459a7c3aa33abb6-00000000000000000000000000000000
I20260927 21:02:16.585942 23263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:02:16.586630 23707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:02:16.587641 23707 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: tablet copy: Starting download of 0 data blocks...
I20260927 21:02:16.587821 23707 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: tablet copy: Starting download of 1 WAL segments...
I20260927 21:02:16.588855 23707 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:02:16.589767 23263 tablet_copy_service.cc:342] P 66b0334abc594ee6832dd56fe5e8d268: Request end of tablet copy session e50e1690dddf4d568459a7c3aa33abb6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:50978
I20260927 21:02:16.589841 23263 tablet_copy_service.cc:433] P 66b0334abc594ee6832dd56fe5e8d268: ending tablet copy session e50e1690dddf4d568459a7c3aa33abb6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e50e1690dddf4d568459a7c3aa33abb6
I20260927 21:02:16.592325 23707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:16.593561 23814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.593765 23707 server_base.cc:1034] running on GCE node
W20260927 21:02:16.593631 23815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.593770 23817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.594077 23707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.594230 23707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.595359 23707 hybrid_clock.cc:648] HybridClock initialized: now 1790542936595349 us; error 24 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.596892 23707 init.cc:377] Logged in from keytab as oryx/127.16.190.253@KRBTEST.COM (short username oryx)
I20260927 21:02:16.597203 23707 webserver.cc:460] Webserver started at http://127.16.190.253:38821/ using document root <none> and password file <none>
I20260927 21:02:16.597327 23707 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.597388 23707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.598083 23707 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:16.598703 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.598860 23707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:16.598915 23707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "e50e1690dddf4d568459a7c3aa33abb6"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "d01b552d220ff7608444c5541c1344ef"
server_key_iv: "404eb8886aa1c40586912d5fda4274ce"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.598975 23707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.626482 23707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.628641 23707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.631291 23707 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.253:34651
I20260927 21:02:16.631331 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.253:34651 every 8 connection(s)
I20260927 21:02:16.631626 23707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
I20260927 21:02:16.632341 23877 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:02:16.632802 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Bootstrap starting.
I20260927 21:02:16.635284 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:02:16.635521 23877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Bootstrap complete.
I20260927 21:02:16.635774 23877 raft_consensus.cc:348] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } }
I20260927 21:02:16.635855 23877 raft_consensus.cc:729] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e50e1690dddf4d568459a7c3aa33abb6, State: Initialized, Role: LEARNER
I20260927 21:02:16.635883 23877 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } }
I20260927 21:02:16.636087 23877 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:16.636081 23880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } }
I20260927 21:02:16.636195 23880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:02:16.637601 23891 catalog_manager.cc:1259] Loaded cluster ID: e5d5347804c04598b3ce8fd71157fd33
I20260927 21:02:16.637663 23891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: loading cluster ID for follower catalog manager: success
I20260927 21:02:16.638607 23891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: acquiring CA information for follower catalog manager: success
I20260927 21:02:16.638890 23891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:02:16.683235 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.676680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.16.190.253:34651)
0927 21:02:16.676851 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.676853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.676862 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:16.676895 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.676896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.676906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.676984 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.677609 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.678272 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.678991 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.679191 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.679272 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.680506 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.680507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.680507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.680515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.682052 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.682418 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.682419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.682420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.682486 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.682690 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.682691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.682691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.682806 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.682918 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.683028 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.683114 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.16.190.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.252.keytab.
Entry for principal oryx/127.16.190.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.252.keytab.
I20260927 21:02:16.712054 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:41353
--webserver_interface=127.16.190.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.252.keytab
--principal=oryx/127.16.190.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:16.795104 23895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:16.795372 23895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:16.795410 23895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:16.796700 23895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:16.796770 23895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:16.796794 23895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:16.796816 23895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:16.796896 23895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:16.798590 23895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.252.keytab
--principal=oryx/127.16.190.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:41353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.190.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:16.798915 23895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:16.799185 23895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:16.804495 23900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.804525 23901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.804458 23903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.804622 23895 server_base.cc:1034] running on GCE node
I20260927 21:02:16.805048 23895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.805362 23895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.806538 23895 hybrid_clock.cc:648] HybridClock initialized: now 1790542936806532 us; error 43 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:16.809374 23895 init.cc:377] Logged in from keytab as oryx/127.16.190.252@KRBTEST.COM (short username oryx)
I20260927 21:02:16.809918 23895 webserver.cc:460] Webserver started at http://127.16.190.252:36847/ using document root <none> and password file <none>
I20260927 21:02:16.810181 23895 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.810256 23895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.810417 23895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:16.811678 23895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/instance:
uuid: "a88b411d25b041eeb1430b20aa493be5"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "2e309143113b9d1c14630540c430c23d"
server_key_iv: "9f659416ef599bc1e5714540fc460e84"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.812074 23895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal/instance:
uuid: "a88b411d25b041eeb1430b20aa493be5"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "2e309143113b9d1c14630540c430c23d"
server_key_iv: "9f659416ef599bc1e5714540fc460e84"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.813598 23895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:16.814409 23910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.814595 23895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:16.814706 23895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "a88b411d25b041eeb1430b20aa493be5"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "2e309143113b9d1c14630540c430c23d"
server_key_iv: "9f659416ef599bc1e5714540fc460e84"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.814795 23895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.825712 23895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:16.828001 23895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:16.828191 23895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:16.833599 23895 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.252:41353
I20260927 21:02:16.833853 23962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.252:41353 every 8 connection(s)
I20260927 21:02:16.834124 23895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
I20260927 21:02:16.835932 23963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:16.837139 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23895
I20260927 21:02:16.838449 23963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.252@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
I20260927 21:02:16.850445 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.837874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.16.190.252:41353)
0927 21:02:16.838030 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.838039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.838069 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:02:16.838643 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.838653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.838721 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.838935 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.839797 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.840880 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.841749 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.842448 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.842580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.846499 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.846505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.846506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.846512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.848620 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.849123 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.849127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.849128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.849229 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.849454 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.849454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.849455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.849675 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.849831 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.849960 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.850180 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260927 21:02:16.850975 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.839315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.16.190.254:37891)
0927 21:02:16.839358 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.839360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.839530 (+ 170us) server_negotiation.cc:407] Connection header received
0927 21:02:16.839609 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.839610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.839620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.839669 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.840271 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.841265 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.842273 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.842478 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.842558 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.847126 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.847128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.847128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.847136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.849216 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.849939 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.849941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.849941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.850005 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.850240 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.850241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.850242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.850377 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.850449 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.850721 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.850827 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:02:16.851461 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.839245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:50994)
0927 21:02:16.839413 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.839459 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.839547 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.839694 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.839697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.840058 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.840162 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.840167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.841393 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.841394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.842176 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.842179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.842644 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.842645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.842730 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.843442 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.843453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.847018 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.849488 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.849495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.849498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.849825 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.850090 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.850091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.850092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.850158 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.850457 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.850463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.850528 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.851141 (+ 613us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.851289 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260927 21:02:16.852559 23963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.252@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
I20260927 21:02:16.860388 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.852957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.16.190.253:34651)
0927 21:02:16.853006 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.853008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.853065 (+ 57us) server_negotiation.cc:407] Connection header received
0927 21:02:16.853154 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.853155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.853165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.853212 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.853563 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.854058 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.855382 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.855579 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.855627 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.857762 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.857764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.857764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.857773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.859191 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.859565 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.859567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.859567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.859630 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.859843 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.859844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.859844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.859962 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.860041 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.860195 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.860287 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:16.860388 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.852881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36842)
0927 21:02:16.853010 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.853025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.853084 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.853221 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.853222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.853411 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.853485 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.853487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.854158 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.854160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.855278 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.855281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.855664 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.855665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.855762 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.856168 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.856174 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.857658 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.859303 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.859304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.859305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.859483 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.859707 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.859708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.859708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.859774 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.860032 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.860033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.860079 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.860168 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.860284 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 21:02:16.861277 23963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.252@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
I20260927 21:02:16.868057 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.861580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.0.0.1:58726)
0927 21:02:16.861632 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.861644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.861682 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.861785 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.861786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.861926 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.861957 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.861959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.862617 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.862618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.863260 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.863262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.863630 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.863631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.863678 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.864198 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.864203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.865465 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.867045 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.867046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.867046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.867209 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.867414 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.867415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.867416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.867467 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.867724 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.867725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.867762 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.867831 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.867931 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:02:16.868060 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.861666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.16.190.252:41353)
0927 21:02:16.861703 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.861705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.861712 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:16.861730 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.861730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.861739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.861784 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.862034 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.862513 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.863324 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.863545 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.863624 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.865556 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.865558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.865558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.865565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.866946 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.867295 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.867296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.867296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.867350 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.867554 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.867555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.867555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.867660 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.867732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.867859 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.867982 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":21}
I20260927 21:02:16.869735 23963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Bootstrap starting.
I20260927 21:02:16.870579 23963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:16.870872 23963 log.cc:824] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:16.871683 23963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: No bootstrap required, opened a new log
I20260927 21:02:16.873214 23963 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } }
I20260927 21:02:16.873351 23963 raft_consensus.cc:374] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:16.873373 23963 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88b411d25b041eeb1430b20aa493be5, State: Initialized, Role: FOLLOWER
I20260927 21:02:16.873559 23963 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } }
I20260927 21:02:16.873886 23963 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:16.873867 23968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } } }
I20260927 21:02:16.874050 23968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:02:16.876124 23979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:02:16.876256 23979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:02:16.879004 23895 master_runner.cc:428] Detected that this master a88b411d25b041eeb1430b20aa493be5 is joining an existing cluster
I20260927 21:02:16.879068 23895 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.190.252:41353
I20260927 21:02:16.889098 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.16.190.253:34651)
0927 21:02:16.880705 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.880707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.881093 (+ 386us) server_negotiation.cc:407] Connection header received
0927 21:02:16.881204 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.881206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.881218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.881252 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.881821 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.882688 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.884004 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.884363 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.884414 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.885794 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.885796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.885796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.885805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.887638 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.888080 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.888081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.888081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.888144 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.888368 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.888369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.888369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.888506 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.888553 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.888772 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.888936 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 21:02:16.889900 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36856)
0927 21:02:16.881023 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.881035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.881122 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.881333 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.881334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.881623 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.881711 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.881714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.882804 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.882805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.883888 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.883891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.884517 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.884518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.884589 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.885183 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.885190 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.885681 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.887740 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.887741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.887741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.887969 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.888224 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.888225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.888225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.888289 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.888604 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.888605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.888653 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.889539 (+ 886us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.889721 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":97,"threads_started":1}
I20260927 21:02:16.891472 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.16.190.254:37891)
0927 21:02:16.880656 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.880659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.881095 (+ 436us) server_negotiation.cc:407] Connection header received
0927 21:02:16.881215 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.881217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.881230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.881266 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.881879 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.882854 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.884342 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.885196 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.885275 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.887191 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.887193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.887193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.887202 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.889305 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.889847 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.889848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.889849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.889913 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.890142 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.890143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.890143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.890299 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.890349 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.891218 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.891321 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:02:16.891920 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:50998)
0927 21:02:16.881014 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.881028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.881127 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.881342 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.881343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.881692 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.881773 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.881776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.882966 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.882967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.884213 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.884216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.885343 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.885344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.885444 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.885962 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.885968 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.887059 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.889419 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.889420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.889420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.889750 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.889996 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.889997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.889998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.890064 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.891014 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.891015 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.891066 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.891600 (+ 534us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.891746 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":103,"threads_started":1}
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542935, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.254@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
I20260927 21:02:16.901903 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23305
I20260927 21:02:16.908650 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:43739
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:37891
--webserver_port=41377
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:02:16.949361 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.894358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.0.0.1:58738)
0927 21:02:16.894417 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.894430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.894484 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.894829 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.894830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.895048 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.895108 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.895110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.896357 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.896358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.897526 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.897533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.898122 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.898123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.898216 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.898768 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.898775 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.900712 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.948162 (+ 47450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.948164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.948418 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.948650 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.948651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.948716 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.949029 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.949030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.949079 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.949144 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.949255 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:02:16.949376 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.0.0.1:58732)
0927 21:02:16.888932 (+ 8072us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.888946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.889028 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.889248 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.889249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.889443 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.889908 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.889911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.890692 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.890692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.891204 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.891205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.894576 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.894577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.894653 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.945454 (+ 50801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.945462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.945982 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.948212 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948212 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.948213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.948415 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.948626 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.948627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.948683 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.949023 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.949023 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.949059 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.949121 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.949248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8050,"thread_start_us":76,"threads_started":1}
I20260927 21:02:16.949775 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.880739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.16.190.252:41353)
0927 21:02:16.880785 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.880787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.889003 (+ 8216us) server_negotiation.cc:407] Connection header received
0927 21:02:16.889095 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.889096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.889105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.889135 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.890014 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.890618 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.891310 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.891595 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.891668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.946161 (+ 54493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.946165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.946165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.946174 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.948133 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948492 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.948493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.948493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.948557 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948745 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.948746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.948762 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.948935 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.948999 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.949587 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.949673 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:02:16.950567 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.894427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.16.190.252:41353)
0927 21:02:16.894642 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.894644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.894654 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:16.894699 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.894700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.894713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.895188 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.895233 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.896254 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.897650 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.898027 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.898104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.900846 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.900848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.900848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.900855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.948023 (+ 47168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948492 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.948493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.948493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.948574 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.948795 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.948796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.948796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.948935 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.948999 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.950398 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.950487 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260927 21:02:16.951023 23243 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.190.252:41353: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a88b411d25b041eeb1430b20aa493be5"
member_type: NON_VOTER
last_known_addr {
host: "127.16.190.252"
port: 41353
}
attrs {
promote: true
}
}
dest_uuid: "66b0334abc594ee6832dd56fe5e8d268"
cas_config_opid_index: 5
I20260927 21:02:16.951309 23243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } }
I20260927 21:02:16.952234 23936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:16.953320 23936 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:02:16.953475 23803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a88b411d25b041eeb1430b20aa493be5 (127.16.190.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.954432 23803 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.190.252:41353
I20260927 21:02:16.954510 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.954595 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:16.954751 23895 master.cc:537] Master@127.16.190.252:41353 shutting down...
I20260927 21:02:16.954780 23803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 21:02:16.955478 23230 consensus_peers.cc:487] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 -> Peer a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353): Couldn't send request to peer a88b411d25b041eeb1430b20aa493be5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:02:16.957019 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.951798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36872)
0927 21:02:16.951977 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.951989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.952035 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.952278 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.952279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.952281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:16.952332 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.952334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.953358 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.953360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.954493 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.954496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.956704 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.956705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.956772 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.956833 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.956889 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.956948 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260927 21:02:16.957285 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.951896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.16.190.253:34651)
0927 21:02:16.951943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.951946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.952095 (+ 149us) server_negotiation.cc:407] Connection header received
0927 21:02:16.952128 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.952129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.952156 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.952216 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:16.952415 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.953182 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.956074 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.956603 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.956704 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.956788 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.957131 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.957190 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:02:16.957803 23851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 LEARNER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 21:02:16.958029 23803 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 21:02:16.959156 23851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a88b411d25b041eeb1430b20aa493be5 (127.16.190.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.960858 23880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b0334abc594ee6832dd56fe5e8d268. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.960975 23880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:02:16.961018 23880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: NON_VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: true } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.961222 23880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:02:16.961257 23803 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: attempting to promote NON_VOTER e50e1690dddf4d568459a7c3aa33abb6 to VOTER
I20260927 21:02:16.961513 23803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } }
I20260927 21:02:16.961946 23895 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:16.962042 23895 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:16.962092 23895 tablet_replica.cc:330] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: stopping tablet replica
I20260927 21:02:16.962903 23851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 LEARNER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 21:02:16.963349 23802 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 21:02:16.963528 23230 consensus_peers.cc:487] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 -> Peer a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353): Couldn't send request to peer a88b411d25b041eeb1430b20aa493be5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 21:02:16.964709 23803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.964879 23851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.965764 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.965843 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:16.965891 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.965945 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:16.966120 23991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:16.966231 23991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:02:16.975898 23895 master.cc:552] Master@127.16.190.252:41353 shutdown complete.
I20260927 21:02:16.975975 23895 master_runner.cc:305] Clearing existing system tablet
I20260927 21:02:16.976140 23895 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:16.976209 23895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:16.976980 23895 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:02:16.977571 23994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:16.977793 23895 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:16.977926 23895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:16.977970 23895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "a88b411d25b041eeb1430b20aa493be5"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "2e309143113b9d1c14630540c430c23d"
server_key_iv: "9f659416ef599bc1e5714540fc460e84"
server_key_version: "encryptionkey@0"
I20260927 21:02:16.978669 23895 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:16.980562 23895 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 21:02:16.980651 23895 log.cc:1196] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 21:02:16.980890 23895 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Deleting consensus metadata
I20260927 21:02:16.981046 23895 master_runner.cc:315] Copying system tablet from 127.16.190.254:37891
I20260927 21:02:16.981719 23895 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.190.254:37891
I20260927 21:02:16.987936 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.982270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.16.190.254:37891)
0927 21:02:16.982316 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:16.982318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:16.982328 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:16.982355 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:16.982355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:16.982365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:16.982421 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:16.982754 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.983260 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.983979 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.984190 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.984278 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.985306 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:16.985308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:16.985308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:16.985316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:16.986769 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.987128 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.987129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.987130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.987191 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:16.987399 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:16.987400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:16.987400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:16.987516 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:16.987587 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:16.987752 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:02:16.987829 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:02:16.987937 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:16.982087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51014)
0927 21:02:16.982206 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:16.982219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:16.982280 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:16.982442 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:16.982443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:16.982624 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:16.982667 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.982669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.983347 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.983348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.983886 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:16.983888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:16.984281 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:16.984281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:16.984342 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:16.984877 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:16.984883 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:16.985206 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:16.986873 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.986874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.986875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.987054 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.987259 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:16.987260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:16.987260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:16.987320 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:16.987589 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:16.987590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:16.987636 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:02:16.987716 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:02:16.987831 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260927 21:02:16.988549 23263 tablet_copy_service.cc:140] P 66b0334abc594ee6832dd56fe5e8d268: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a88b411d25b041eeb1430b20aa493be5 ({username='oryx', principal='oryx/127.16.190.252@KRBTEST.COM'} at 127.0.0.1:51014)
I20260927 21:02:16.988633 23263 tablet_copy_service.cc:161] P 66b0334abc594ee6832dd56fe5e8d268: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a88b411d25b041eeb1430b20aa493be5 at {username='oryx', principal='oryx/127.16.190.252@KRBTEST.COM'} at 127.0.0.1:51014: session id = a88b411d25b041eeb1430b20aa493be5-00000000000000000000000000000000
I20260927 21:02:16.989295 23263 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 21:02:16.989837 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 21:02:16.990829 23895 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: tablet copy: Starting download of 0 data blocks...
I20260927 21:02:16.990996 23895 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: tablet copy: Starting download of 1 WAL segments...
I20260927 21:02:16.991762 23895 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 21:02:16.992529 23263 tablet_copy_service.cc:342] P 66b0334abc594ee6832dd56fe5e8d268: Request end of tablet copy session a88b411d25b041eeb1430b20aa493be5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.190.252@KRBTEST.COM'} at 127.0.0.1:51014
I20260927 21:02:16.992607 23263 tablet_copy_service.cc:433] P 66b0334abc594ee6832dd56fe5e8d268: ending tablet copy session a88b411d25b041eeb1430b20aa493be5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a88b411d25b041eeb1430b20aa493be5
I20260927 21:02:16.994990 23895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:16.996726 23895 server_base.cc:1034] running on GCE node
W20260927 21:02:16.996795 24001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.997026 24004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:16.997231 24002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:16.997448 23895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:16.997643 23895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:16.998754 23895 hybrid_clock.cc:648] HybridClock initialized: now 1790542936998744 us; error 26 us; skew 500 ppm
Sep 27 21:02:16 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542936, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:17.000348 23895 init.cc:377] Logged in from keytab as oryx/127.16.190.252@KRBTEST.COM (short username oryx)
I20260927 21:02:17.000638 23895 webserver.cc:460] Webserver started at http://127.16.190.252:44791/ using document root <none> and password file <none>
I20260927 21:02:17.000782 23895 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:17.000852 23895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:17.001515 23895 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:17.002091 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.002249 23895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:17.002295 23895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "a88b411d25b041eeb1430b20aa493be5"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "2e309143113b9d1c14630540c430c23d"
server_key_iv: "9f659416ef599bc1e5714540fc460e84"
server_key_version: "encryptionkey@0"
I20260927 21:02:17.002357 23895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:17.020134 23895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:17.022110 23895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:17.025358 23895 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.252:41353
I20260927 21:02:17.025780 23895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
I20260927 21:02:17.026055 24063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.252:41353 every 8 connection(s)
I20260927 21:02:17.026748 24064 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:02:17.027154 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Bootstrap starting.
I20260927 21:02:17.029862 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:02:17.030089 24064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: Bootstrap complete.
I20260927 21:02:17.030299 24064 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } }
I20260927 21:02:17.030392 24064 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a88b411d25b041eeb1430b20aa493be5, State: Initialized, Role: LEARNER
I20260927 21:02:17.030437 24064 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } }
I20260927 21:02:17.030608 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:17.030735 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:02:17.030661 24064 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:17.032145 24078 catalog_manager.cc:1259] Loaded cluster ID: e5d5347804c04598b3ce8fd71157fd33
I20260927 21:02:17.032234 24078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: loading cluster ID for follower catalog manager: success
I20260927 21:02:17.033411 24078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: acquiring CA information for follower catalog manager: success
I20260927 21:02:17.033710 24078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 21:02:17.040860 23989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:17.041175 23989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:17.041214 23989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:17.043208 23989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:17.043299 23989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:17.043354 23989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:17.045800 23989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=41377
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:17.046113 23989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:17.046413 23989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:17.052649 24083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:17.052652 24084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:17.052652 24086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:17.052846 23989 server_base.cc:1034] running on GCE node
I20260927 21:02:17.053167 23989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:17.053467 23989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:17.054658 23989 hybrid_clock.cc:648] HybridClock initialized: now 1790542937054642 us; error 31 us; skew 500 ppm
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:17.058177 23989 init.cc:377] Logged in from keytab as oryx/127.16.190.193@KRBTEST.COM (short username oryx)
I20260927 21:02:17.058792 23989 webserver.cc:460] Webserver started at http://127.16.190.193:41377/ using document root <none> and password file <none>
I20260927 21:02:17.059093 23989 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:17.059185 23989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:17.061757 23989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:02:17.062567 24093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.062718 23989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:17.062804 23989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "9f98770f6825405a98b88d0780517753"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "e3c4702581ca33ea4d0cb178569188d6"
server_key_iv: "a468481555ff484f0876cd8b71e07162"
server_key_version: "encryptionkey@0"
I20260927 21:02:17.063266 23989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:17.089951 23989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:17.092020 23989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:17.092137 23989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:17.092304 23989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:17.092656 23989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:17.092717 23989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.092759 23989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:17.092787 23989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.098909 23989 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:43739
I20260927 21:02:17.098953 24206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:43739 every 8 connection(s)
I20260927 21:02:17.099309 23989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:17.103404 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 23989
I20260927 21:02:17.103585 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23439
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:17.111346 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:41531
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:37891
--webserver_port=37687
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:02:17.113029 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:46653 (local address 127.16.190.253:34651)
0927 21:02:17.100458 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.100460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.100567 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:02:17.101557 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.101559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.101571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.101604 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.102463 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103233 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105521 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105832 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105879 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.108917 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.108919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.108919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.108928 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.110629 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.111601 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.111602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.111603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.111905 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112225 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.112226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.112227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.112394 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.112446 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.112847 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.112932 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 21:02:17.113282 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.16.190.193:46653)
0927 21:02:17.100515 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.100524 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.101445 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.101714 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.101715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.102139 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.102351 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.102357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103374 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.103374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.105404 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.105408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105978 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.106135 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.106843 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.106850 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.108772 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.110715 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.110723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.110726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.111512 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.111983 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.111984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.111984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.112147 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.112478 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.112484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.112677 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.112784 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.112997 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":861,"thread_start_us":59,"threads_started":1}
I20260927 21:02:17.113768 24209 heartbeater.cc:347] Connected to a master server at 127.16.190.253:34651
I20260927 21:02:17.113889 24209 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.114130 24209 heartbeater.cc:511] Master 127.16.190.253:34651 requested a full tablet report, sending...
I20260927 21:02:17.114144 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:49225 (local address 127.16.190.252:41353)
0927 21:02:17.100618 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.100620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.100627 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:17.101534 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.101535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.101549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.101627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.102441 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103294 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105511 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105871 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105934 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.110867 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.110869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.110870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.110878 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.112572 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112988 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.112989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.112990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.113044 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.113297 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.113298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.113299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.113443 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.113934 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.113992 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.114079 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":56,"threads_started":1}
I20260927 21:02:17.114753 23841 ts_manager.cc:194] Registered new tserver with Master: 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739)
I20260927 21:02:17.115278 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.16.190.193:49225)
0927 21:02:17.100246 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.100290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.101418 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.101708 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.101712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.102124 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.102349 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.102354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103392 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.103393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.105409 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.105413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105990 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.106135 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.106809 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.106820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.110763 (+ 3943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.112654 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.112655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.112909 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.113114 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.113114 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.113115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.113176 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.113599 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.113600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.113647 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.113710 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.113824 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":568,"thread_start_us":95,"threads_started":1}
I20260927 21:02:17.115729 24208 heartbeater.cc:347] Connected to a master server at 127.16.190.252:41353
I20260927 21:02:17.115816 24208 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.115906 24208 heartbeater.cc:511] Master 127.16.190.252:41353 requested a full tablet report, sending...
I20260927 21:02:17.115936 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:40105 (local address 127.16.190.254:37891)
0927 21:02:17.100405 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.100407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.100420 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:02:17.101557 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.101558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.101571 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.101604 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.102463 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103236 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105522 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105833 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105879 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.110079 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.110081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.110082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.110091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.111848 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112591 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.112592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.112593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.112653 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112906 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.112907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.112908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.113052 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.113108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.115698 (+ 2590us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.115813 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260927 21:02:17.116290 24028 ts_manager.cc:194] Registered new tserver with Master: 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739)
I20260927 21:02:17.116350 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.100069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:40105)
0927 21:02:17.100247 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.100290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.101445 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.101730 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.101730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.102178 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.102351 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.102357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.103374 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.103374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.105405 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.105408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.105978 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.105979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.106135 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.107260 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.107265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.109921 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.112273 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.112274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.112495 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.112737 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.112737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.112738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.112789 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.115496 (+ 2707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.115497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.115548 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.116054 (+ 506us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.116192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":118,"thread_start_us":93,"threads_started":1}
I20260927 21:02:17.116722 24207 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:17.116770 24207 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.116868 24207 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:17.117143 23243 ts_manager.cc:194] Re-registered known tserver with Master: 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739)
I20260927 21:02:17.117733 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:40105
W20260927 21:02:17.198370 24215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:17.198603 24215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:17.198629 24215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:17.199816 24215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:17.199862 24215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:17.199888 24215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:17.201429 24215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:41531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=37687
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:17.201642 24215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:17.201817 24215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:17.207109 24220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:17.207175 24215 server_base.cc:1034] running on GCE node
W20260927 21:02:17.207155 24221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:17.207142 24223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:17.207540 24215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:17.207818 24215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:17.209024 24215 hybrid_clock.cc:648] HybridClock initialized: now 1790542937209016 us; error 38 us; skew 500 ppm
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:17.211628 24215 init.cc:377] Logged in from keytab as oryx/127.16.190.194@KRBTEST.COM (short username oryx)
I20260927 21:02:17.212070 24215 webserver.cc:460] Webserver started at http://127.16.190.194:37687/ using document root <none> and password file <none>
I20260927 21:02:17.212323 24215 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:17.212409 24215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:17.213867 24215 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:17.214496 24230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.214722 24215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:17.214826 24215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "693503666d254f348932e12a588a1448"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "434b505e988fa3c6a9ad2b10c8b27c7d"
server_key_iv: "0b884ec972ffe6be8b6a85bb34f21e44"
server_key_version: "encryptionkey@0"
I20260927 21:02:17.215181 24215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:17.233202 24215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:17.235575 24215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:17.235759 24215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:17.236070 24215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:17.236478 24215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:17.236513 24215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.236568 24215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:17.236617 24215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.243116 24215 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:41531
I20260927 21:02:17.243158 24345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:41531 every 8 connection(s)
I20260927 21:02:17.243544 24215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:17.248090 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 24215
I20260927 21:02:17.248281 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23573
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
I20260927 21:02:17.254549 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:35689
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:37891
--webserver_port=35373
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:17.255683 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:48199 (local address 127.16.190.252:41353)
0927 21:02:17.244794 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.244797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.244876 (+ 79us) server_negotiation.cc:407] Connection header received
0927 21:02:17.245508 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.245509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.245520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.245576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.246446 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.246954 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.249132 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.249434 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.249478 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.252254 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.252255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.252256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.252264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.253976 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254495 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.254496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.254497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.254556 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254872 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.254873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.254873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.255018 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.255061 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.255484 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.255582 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:17.255713 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:53615 (local address 127.16.190.253:34651)
0927 21:02:17.244368 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.244370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.244664 (+ 294us) server_negotiation.cc:407] Connection header received
0927 21:02:17.245379 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.245381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.245395 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.245447 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.246461 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.247231 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.248514 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.248740 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.248780 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.252662 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.252663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.252664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.252672 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.254272 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254752 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.254753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.254754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.254816 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.255024 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.255025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.255025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.255167 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.255210 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.255466 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.255566 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:02:17.256389 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.16.190.194:53615)
0927 21:02:17.244522 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.244570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.245281 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.245870 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.245871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.246137 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.246341 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.246348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.247305 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.247305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.248415 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.248418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.248919 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.248920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.249068 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.249697 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.249707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.252557 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.254367 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.254369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.254640 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.254893 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.254894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.254957 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.255256 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.255257 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.255326 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.255852 (+ 526us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.256126 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":123,"thread_start_us":95,"threads_started":1}
I20260927 21:02:17.256394 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.16.190.194:48199)
0927 21:02:17.244795 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.244806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.245254 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.245690 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.245696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.246141 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.246340 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.246345 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.247070 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.247071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.249041 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.249043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.249566 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.249567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.249657 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.250099 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.250104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.252146 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.254073 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.254085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.254407 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.254629 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.254630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.254630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.254789 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.255128 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.255135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.255320 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.255817 (+ 497us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.256027 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":7,"thread_start_us":78,"threads_started":1}
I20260927 21:02:17.256901 24348 heartbeater.cc:347] Connected to a master server at 127.16.190.253:34651
I20260927 21:02:17.257004 24348 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.257018 24347 heartbeater.cc:347] Connected to a master server at 127.16.190.252:41353
I20260927 21:02:17.257076 24347 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.257205 24348 heartbeater.cc:511] Master 127.16.190.253:34651 requested a full tablet report, sending...
I20260927 21:02:17.257200 24347 heartbeater.cc:511] Master 127.16.190.252:41353 requested a full tablet report, sending...
I20260927 21:02:17.257750 23841 ts_manager.cc:194] Registered new tserver with Master: 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:17.257766 24028 ts_manager.cc:194] Registered new tserver with Master: 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:17.259096 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:47503 (local address 127.16.190.254:37891)
0927 21:02:17.244953 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.244955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.244964 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:17.245629 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.245631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.245642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.245695 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.246460 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.247219 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.251069 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.251369 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.251444 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.255865 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.255866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.255867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.255875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.257871 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.258288 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.258289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.258289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.258346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.258591 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.258592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.258592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.258694 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.258748 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.258920 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.258985 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:17.259382 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.244323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:47503)
0927 21:02:17.244522 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.244570 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.245273 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.245730 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.245731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.246158 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.246341 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.246347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.250248 (+ 3901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.250249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.250973 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.250976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.251503 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.251505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.251592 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.252567 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.252572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.255748 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.257964 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.257965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.257966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.258201 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.258437 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.258437 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.258438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.258511 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.258758 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.258759 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.258822 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.258896 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.259019 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":30,"spinlock_wait_cycles":7936,"thread_start_us":95,"threads_started":1}
I20260927 21:02:17.259817 24346 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:17.259872 24346 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.259949 24346 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:17.260226 23243 ts_manager.cc:194] Re-registered known tserver with Master: 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:17.260608 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:47503
W20260927 21:02:17.342025 24353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:17.342290 24353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:17.342317 24353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:17.343444 24353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:17.343484 24353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:17.343510 24353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:17.344985 24353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=35373
--tserver_master_addrs=127.16.190.254:37891,127.16.190.253:34651,127.16.190.252:41353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:17.345186 24353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:17.345366 24353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:17.350499 24358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:17.350584 24361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:17.350484 24359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:17.350848 24353 server_base.cc:1034] running on GCE node
I20260927 21:02:17.351075 24353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:17.351337 24353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:17.352502 24353 hybrid_clock.cc:648] HybridClock initialized: now 1790542937352487 us; error 29 us; skew 500 ppm
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:17.359795 24353 init.cc:377] Logged in from keytab as oryx/127.16.190.195@KRBTEST.COM (short username oryx)
I20260927 21:02:17.360345 24353 webserver.cc:460] Webserver started at http://127.16.190.195:35373/ using document root <none> and password file <none>
I20260927 21:02:17.360601 24353 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:17.360692 24353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:17.362264 24353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:17.362957 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.363227 24353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:17.363341 24353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3e2213654f514bfbb47eb3b9b0ff9efa"
format_stamp: "Formatted at 2026-09-27 21:02:16 on dist-test-slave-6c7s"
server_key: "a5a86b245996adc3278c69bb43e94a28"
server_key_iv: "9b833fcf82bb5c9294ebcfa9c0f3eb78"
server_key_version: "encryptionkey@0"
I20260927 21:02:17.363710 24353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:17.385200 24353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:17.387333 24353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:17.387502 24353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:17.387738 24353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:17.388055 24353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:17.388111 24353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.388155 24353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:17.388188 24353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:17.394599 24353 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:35689
I20260927 21:02:17.395010 24353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:17.395493 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.392210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.0.0.1:58748)
0927 21:02:17.392252 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.392265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.392317 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.392554 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.392554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.392556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.392615 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.392618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.393673 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.393674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.394493 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.394496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.395196 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.395198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.395280 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.395286 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.395350 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.395403 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 21:02:17.395593 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.392293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.16.190.252:41353)
0927 21:02:17.392349 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.392351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.392364 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:02:17.392403 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.392404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.392431 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.392524 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.392678 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.393541 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.394604 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.395082 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.395168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.395247 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.395481 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.395525 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:02:17.395844 24038 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 21:02:17.396050 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b0334abc594ee6832dd56fe5e8d268. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:17.396154 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: LEARNER
I20260927 21:02:17.395375 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:35689 every 8 connection(s)
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.252@KRBTEST.COM
I20260927 21:02:17.403291 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 24353
Sep 27 21:02:17 dist-test-slave-6c7s krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542937, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.253@KRBTEST.COM
I20260927 21:02:17.407778 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.396109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:59181 (local address 127.16.190.254:37891)
0927 21:02:17.396159 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.396161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.396616 (+ 455us) server_negotiation.cc:407] Connection header received
0927 21:02:17.397127 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.397127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.397138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.397183 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.398082 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.398864 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.399965 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400249 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400315 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.403164 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.403165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.403166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.403173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.405313 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.406475 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.406476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.406477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.406538 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.406825 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.406827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.406827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.406968 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.407029 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.407592 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.407671 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:17.408295 24488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.395913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:59181)
0927 21:02:17.396534 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.396543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.396981 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.397593 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.397594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.397777 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.397972 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.397977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.398961 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.398962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.399880 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.399883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400338 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.400485 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.401346 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.401352 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.403048 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.405793 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.405802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.405805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.406381 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.406607 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.406608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.406609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.406745 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.407056 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.407063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.407258 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.407378 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.407936 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":48,"thread_start_us":51,"threads_started":1}
I20260927 21:02:17.408528 24489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.396638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.16.190.195:52997)
0927 21:02:17.396888 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.396899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.396971 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.397209 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.397213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.397739 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.397972 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.397978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.399007 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.399008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.399947 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.399948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400433 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.400500 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.401273 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.401286 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.404997 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.407419 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.407419 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.407420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.407626 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.407841 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.407841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.407842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.407901 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.408223 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.408224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.408264 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.408343 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.408449 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":30,"thread_start_us":109,"threads_started":1}
I20260927 21:02:17.408849 24483 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37891
I20260927 21:02:17.408885 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.396803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:52997 (local address 127.16.190.253:34651)
0927 21:02:17.396866 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.396868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.396948 (+ 80us) server_negotiation.cc:407] Connection header received
0927 21:02:17.397055 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.397056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.397067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.397111 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.398159 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.398935 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400072 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400374 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400431 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.405094 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.405095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.405096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.405104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.407337 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.407704 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.407705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.407706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.407770 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.407969 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.407970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.407970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.408085 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.408194 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.408659 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.408751 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:17.408962 24483 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.409164 24483 heartbeater.cc:511] Master 127.16.190.254:37891 requested a full tablet report, sending...
I20260927 21:02:17.409332 24485 heartbeater.cc:347] Connected to a master server at 127.16.190.253:34651
I20260927 21:02:17.409381 24485 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.409446 24485 heartbeater.cc:511] Master 127.16.190.253:34651 requested a full tablet report, sending...
I20260927 21:02:17.409689 23243 ts_manager.cc:194] Re-registered known tserver with Master: 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:17.409785 23841 ts_manager.cc:194] Registered new tserver with Master: 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:17.410136 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.396082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:34359 (local address 127.16.190.252:41353)
0927 21:02:17.396116 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.396118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.396332 (+ 214us) server_negotiation.cc:407] Connection header received
0927 21:02:17.397074 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.397075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.397088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.397118 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.398080 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.398689 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400050 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400271 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400332 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.404587 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.404589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.404589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.404596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.406613 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.409303 (+ 2690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.409304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.409312 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.409373 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.409597 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.409598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.409598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.409716 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.409767 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.409954 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.410039 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:02:17.410173 23243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:59181
I20260927 21:02:17.410686 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.395818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.16.190.195:34359)
0927 21:02:17.396188 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.396233 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.396970 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.397209 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.397213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.397747 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:17.397972 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.397978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.398792 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.398793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.399945 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.399948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.400408 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.400408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.400485 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.401556 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:17.401561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:17.404476 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:17.409004 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.409005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.409006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.409216 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.409448 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:17.409449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:17.409449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:17.409525 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:17.409800 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:17.409801 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:17.409839 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.410238 (+ 399us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.410355 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260927 21:02:17.411019 24484 heartbeater.cc:347] Connected to a master server at 127.16.190.252:41353
I20260927 21:02:17.411077 24484 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:17.411145 24484 heartbeater.cc:511] Master 127.16.190.252:41353 requested a full tablet report, sending...
I20260927 21:02:17.411415 24028 ts_manager.cc:194] Registered new tserver with Master: 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:17.414970 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.404387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.16.190.254:37891)
0927 21:02:17.404649 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.404651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.405552 (+ 901us) server_negotiation.cc:407] Connection header received
0927 21:02:17.405667 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.405668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.405682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.405731 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.406738 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.407525 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.409464 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.409878 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.409947 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.411774 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.411776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.411776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.411784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.413679 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.414089 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.414090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.414091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.414151 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.414351 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.414352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.414352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.414486 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.414601 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.414728 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.414813 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":73,"threads_started":1}
I20260927 21:02:17.417518 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.404658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.16.190.253:34651)
0927 21:02:17.405652 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.405655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.405662 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:17.405682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.405683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.405696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.406906 (+ 1210us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.406941 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.407666 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.412398 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.412743 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.412838 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.414224 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.414225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.414225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.414233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.416267 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.416660 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.416661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.416661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.416729 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.416956 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.416957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.416958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.417076 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.417201 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.417318 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.417404 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":53,"threads_started":1}
I20260927 21:02:17.416859 23243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:17.419188 23243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:17.420069 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: NON_VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: true } } }
I20260927 21:02:17.420184 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:17.420531 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.405232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.16.190.252:41353)
0927 21:02:17.405791 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.405793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.405800 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:17.405817 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.405818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.405828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.405885 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.407749 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.408693 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.412200 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.412496 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.412589 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.413908 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.413909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.413909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.413916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.415667 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.416080 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.416081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.416082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.416137 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.419696 (+ 3559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.419697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.419697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.419849 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.419993 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.420334 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.420431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":68,"threads_started":1}
I20260927 21:02:17.421989 23802 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: attempting to promote NON_VOTER a88b411d25b041eeb1430b20aa493be5 to VOTER
I20260927 21:02:17.422163 23802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:17.422696 23851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 21:02:17.422881 24037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 LEARNER]: Refusing update from remote peer 66b0334abc594ee6832dd56fe5e8d268: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 21:02:17.422878 23992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 21:02:17.423072 23992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 21:02:17.423566 24506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a88b411d25b041eeb1430b20aa493be5 (127.16.190.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.423719 23851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a88b411d25b041eeb1430b20aa493be5 (127.16.190.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.424641 23991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.424633 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.424714 23991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:02:17.424714 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:17.424677 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.424764 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:17.424930 23803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.424999 23803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:17.424925 24037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a88b411d25b041eeb1430b20aa493be5 (127.16.190.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.426319 24067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66b0334abc594ee6832dd56fe5e8d268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:17.426429 24067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:02:17.430101 24508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.425615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:43739 (local address 127.0.0.1:36280)
0927 21:02:17.426283 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.426293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.426342 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.426895 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.426896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.426899 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.426952 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.426954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428286 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.428287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.429099 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.429101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.429838 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.429838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.429914 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.429919 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.429971 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.430015 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":90,"threads_started":1}
I20260927 21:02:17.430174 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.425833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.16.190.193:43739)
0927 21:02:17.426230 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.426235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.426396 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:02:17.426608 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.426611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.426629 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.426790 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.427048 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428177 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.429193 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.429752 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.429854 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.429924 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.430027 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.430069 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":79,"threads_started":1}
I20260927 21:02:17.430567 23801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.425572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:41531 (local address 127.0.0.1:49402)
0927 21:02:17.425720 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.425731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.425772 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.427251 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.427252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.427254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.427291 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.427293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428163 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.428165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.429322 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.429324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.430331 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.430332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.430390 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.430393 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.430460 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.430502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 21:02:17.430651 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.425853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.16.190.194:41531)
0927 21:02:17.427088 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.427092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.427100 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:17.427145 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.427146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.427159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.427243 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.427343 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428050 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.429408 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.430260 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.430351 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.430404 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.430474 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.430514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":95,"threads_started":1}
I20260927 21:02:17.431221 24279 tablet_service.cc:1467] Processing CreateTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 (DEFAULT_TABLE table=test-table [id=b031baf6facf4a3ab6414670d2a20c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:17.431221 24141 tablet_service.cc:1467] Processing CreateTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 (DEFAULT_TABLE table=test-table [id=b031baf6facf4a3ab6414670d2a20c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:17.431608 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2632bc1ec2c4f3996e4d2105707cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:17.431608 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2632bc1ec2c4f3996e4d2105707cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:17.435701 24513 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Bootstrap starting.
I20260927 21:02:17.436115 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.426472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.16.190.195:35689)
0927 21:02:17.426744 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.426749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.426760 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:02:17.426801 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.426803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.426820 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.426908 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.427047 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428183 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.429709 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.435680 (+ 5971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.435789 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.435852 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.436005 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.436050 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260927 21:02:17.436162 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.425590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:35689 (local address 127.0.0.1:34202)
0927 21:02:17.426036 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.426047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.426097 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.426916 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.426917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.426918 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.426971 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.426972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.428258 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.428260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.429620 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.429622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.435785 (+ 6163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.435787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.435908 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.435915 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.435984 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.436065 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260927 21:02:17.436508 24513 tablet_bootstrap.cc:654] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:17.436941 24513 log.cc:824] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:17.436975 24417 tablet_service.cc:1467] Processing CreateTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 (DEFAULT_TABLE table=test-table [id=b031baf6facf4a3ab6414670d2a20c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:17.437388 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2632bc1ec2c4f3996e4d2105707cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:17.437372 24514 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Bootstrap starting.
I20260927 21:02:17.437947 24513 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: No bootstrap required, opened a new log
I20260927 21:02:17.437973 24514 tablet_bootstrap.cc:654] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:17.438055 24513 ts_tablet_manager.cc:1397] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:17.438266 24514 log.cc:824] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:17.438961 24514 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: No bootstrap required, opened a new log
I20260927 21:02:17.439054 24514 ts_tablet_manager.cc:1397] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:17.440593 24513 raft_consensus.cc:348] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.440793 24513 raft_consensus.cc:374] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:17.440871 24513 raft_consensus.cc:729] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f98770f6825405a98b88d0780517753, State: Initialized, Role: FOLLOWER
I20260927 21:02:17.441025 24513 consensus_queue.cc:260] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.441334 24207 heartbeater.cc:503] Master 127.16.190.254:37891 was elected leader, sending a full tablet report...
I20260927 21:02:17.441331 24513 ts_tablet_manager.cc:1428] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:02:17.441255 24514 raft_consensus.cc:348] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.441412 24514 raft_consensus.cc:374] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:17.441452 24514 raft_consensus.cc:729] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693503666d254f348932e12a588a1448, State: Initialized, Role: FOLLOWER
I20260927 21:02:17.441583 24514 consensus_queue.cc:260] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.441829 24514 ts_tablet_manager.cc:1428] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:02:17.441946 24346 heartbeater.cc:503] Master 127.16.190.254:37891 was elected leader, sending a full tablet report...
I20260927 21:02:17.442728 24517 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Bootstrap starting.
I20260927 21:02:17.443298 24517 tablet_bootstrap.cc:654] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:17.443590 24517 log.cc:824] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:17.444464 24517 tablet_bootstrap.cc:492] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: No bootstrap required, opened a new log
I20260927 21:02:17.444542 24517 ts_tablet_manager.cc:1397] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:17.446152 24517 raft_consensus.cc:348] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.446323 24517 raft_consensus.cc:374] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:17.446370 24517 raft_consensus.cc:729] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2213654f514bfbb47eb3b9b0ff9efa, State: Initialized, Role: FOLLOWER
I20260927 21:02:17.446491 24517 consensus_queue.cc:260] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.446772 24517 ts_tablet_manager.cc:1428] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:02:17.446844 24483 heartbeater.cc:503] Master 127.16.190.254:37891 was elected leader, sending a full tablet report...
W20260927 21:02:17.494335 24349 tablet.cc:2369] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:17.574396 24518 raft_consensus.cc:482] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:17.574595 24518 raft_consensus.cc:504] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.575049 24518 leader_election.cc:288] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 693503666d254f348932e12a588a1448 (127.16.190.194:41531), 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:17.579648 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.575478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:48613 (local address 127.16.190.194:41531)
0927 21:02:17.575537 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.575539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.575547 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:17.575591 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.575592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.575604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.575676 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.576008 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.576782 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.578562 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.579207 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.579274 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.579282 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.579526 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.579574 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:17.579674 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.575479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:39597 (local address 127.16.190.195:35689)
0927 21:02:17.575529 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.575531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.575575 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:02:17.575649 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.575650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.575660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.575709 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:17.576008 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.576782 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.578609 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.579173 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.579268 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.579277 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.579526 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.579613 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:17.579658 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.575351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:41531 (local address 127.16.190.193:48613)
0927 21:02:17.575422 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.575435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.575487 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.575753 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.575754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.575757 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.575872 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.575874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.577027 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.577028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.578416 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.578420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.579314 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.579315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.579418 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.579425 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.579496 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.579549 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":99}
I20260927 21:02:17.579658 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.575379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:35689 (local address 127.16.190.193:39597)
0927 21:02:17.575508 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:17.575518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:17.575575 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:17.575753 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:17.575754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:17.575757 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:17.575872 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.575874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.577011 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.577013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.578416 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:17.578420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.579314 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.579315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:17.579418 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.579425 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 21:02:17.579496 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:02:17.579549 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":191}
I20260927 21:02:17.580138 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2632bc1ec2c4f3996e4d2105707cac0" candidate_uuid: "9f98770f6825405a98b88d0780517753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693503666d254f348932e12a588a1448" is_pre_election: true
I20260927 21:02:17.580349 24299 raft_consensus.cc:2454] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f98770f6825405a98b88d0780517753 in term 0.
I20260927 21:02:17.580368 24437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2632bc1ec2c4f3996e4d2105707cac0" candidate_uuid: "9f98770f6825405a98b88d0780517753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" is_pre_election: true
I20260927 21:02:17.580564 24437 raft_consensus.cc:2454] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f98770f6825405a98b88d0780517753 in term 0.
I20260927 21:02:17.580641 24094 leader_election.cc:302] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693503666d254f348932e12a588a1448, 9f98770f6825405a98b88d0780517753; no voters:
I20260927 21:02:17.580852 24518 raft_consensus.cc:2781] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:17.580931 24518 raft_consensus.cc:482] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:17.580960 24518 raft_consensus.cc:3037] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:17.581701 24518 raft_consensus.cc:504] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.581873 24518 leader_election.cc:288] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [CANDIDATE]: Term 1 election: Requested vote from peers 693503666d254f348932e12a588a1448 (127.16.190.194:41531), 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689)
I20260927 21:02:17.582034 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2632bc1ec2c4f3996e4d2105707cac0" candidate_uuid: "9f98770f6825405a98b88d0780517753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693503666d254f348932e12a588a1448"
I20260927 21:02:17.582125 24299 raft_consensus.cc:3037] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:17.582072 24437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2632bc1ec2c4f3996e4d2105707cac0" candidate_uuid: "9f98770f6825405a98b88d0780517753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa"
I20260927 21:02:17.582188 24437 raft_consensus.cc:3037] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:17.582857 24299 raft_consensus.cc:2454] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f98770f6825405a98b88d0780517753 in term 1.
I20260927 21:02:17.583045 24094 leader_election.cc:302] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693503666d254f348932e12a588a1448, 9f98770f6825405a98b88d0780517753; no voters:
I20260927 21:02:17.583153 24518 raft_consensus.cc:2781] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:17.583168 24437 raft_consensus.cc:2454] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f98770f6825405a98b88d0780517753 in term 1.
I20260927 21:02:17.583302 24518 raft_consensus.cc:686] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 1 LEADER]: Becoming Leader. State: Replica: 9f98770f6825405a98b88d0780517753, State: Running, Role: LEADER
I20260927 21:02:17.583418 24518 consensus_queue.cc:237] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:17.584334 23243 catalog_manager.cc:5495] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f98770f6825405a98b88d0780517753 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "9f98770f6825405a98b88d0780517753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } health_report { overall_health: UNKNOWN } } }
W20260927 21:02:17.600971 24210 tablet.cc:2369] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:17.609979 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.607336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.16.190.193:43739)
0927 21:02:17.607396 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.607398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.607408 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:17.607480 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.607481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.607499 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.607582 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:17.607756 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.608261 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.609143 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.609340 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.609413 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.609620 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:17.609724 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:17.609823 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.609843 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.609865 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:02:17.611918 24299 raft_consensus.cc:1261] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Refusing update from remote peer 9f98770f6825405a98b88d0780517753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:17.611969 24437 raft_consensus.cc:1261] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Refusing update from remote peer 9f98770f6825405a98b88d0780517753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:17.612342 24518 consensus_queue.cc:1035] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:17.612515 24522 consensus_queue.cc:1035] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:17.614847 24525 mvcc.cc:205] Tried to move back new op lower bound from 7334063872455630848 to 7334063872342429696. Current Snapshot: MvccSnapshot[applied={T|T < 7334063872455630848}]
I20260927 21:02:17.615870 24527 mvcc.cc:205] Tried to move back new op lower bound from 7334063872455630848 to 7334063872342429696. Current Snapshot: MvccSnapshot[applied={T|T < 7334063872455630848}]
I20260927 21:02:17.618180 23242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:17.618297 23242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:17.619797 23242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66b0334abc594ee6832dd56fe5e8d268: Sending DeleteTablet for 3 replicas of tablet c2632bc1ec2c4f3996e4d2105707cac0
I20260927 21:02:17.620149 24279 tablet_service.cc:1514] Processing DeleteTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:17 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:49402
I20260927 21:02:17.620194 24141 tablet_service.cc:1514] Processing DeleteTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:17 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:36280
I20260927 21:02:17.620205 24417 tablet_service.cc:1514] Processing DeleteTablet for tablet c2632bc1ec2c4f3996e4d2105707cac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:17 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:34202
I20260927 21:02:17.620380 24536 tablet_replica.cc:330] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: stopping tablet replica
I20260927 21:02:17.620484 24536 raft_consensus.cc:2229] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:17.620595 24536 raft_consensus.cc:2258] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:17.621235 24536 ts_tablet_manager.cc:1905] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:17.621385 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23211
I20260927 21:02:17.623371 24536 ts_tablet_manager.cc:1918] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:17.623453 24536 log.cc:1196] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/c2632bc1ec2c4f3996e4d2105707cac0
I20260927 21:02:17.623741 24536 ts_tablet_manager.cc:1939] T c2632bc1ec2c4f3996e4d2105707cac0 P 693503666d254f348932e12a588a1448: Deleting consensus metadata
I20260927 21:02:17.624951 24538 tablet_replica.cc:330] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: stopping tablet replica
I20260927 21:02:17.625078 24538 raft_consensus.cc:2229] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:17.625200 24538 raft_consensus.cc:2258] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:17.625320 24537 tablet_replica.cc:330] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: stopping tablet replica
I20260927 21:02:17.625481 24537 raft_consensus.cc:2229] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:17.625627 24537 raft_consensus.cc:2258] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:17.626247 24537 ts_tablet_manager.cc:1905] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:17.628227 24537 ts_tablet_manager.cc:1918] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:17.628300 24537 log.cc:1196] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/c2632bc1ec2c4f3996e4d2105707cac0
I20260927 21:02:17.628547 24537 ts_tablet_manager.cc:1939] T c2632bc1ec2c4f3996e4d2105707cac0 P 9f98770f6825405a98b88d0780517753: Deleting consensus metadata
I20260927 21:02:17.629127 24538 ts_tablet_manager.cc:1905] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:02:17.629328 24096 connection.cc:535] server connection from 127.0.0.1:36280 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:02:17.629531 24234 connection.cc:535] server connection from 127.0.0.1:49402 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:02:17.631016 24538 ts_tablet_manager.cc:1918] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:17.631089 24538 log.cc:1196] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/c2632bc1ec2c4f3996e4d2105707cac0
I20260927 21:02:17.631378 24538 ts_tablet_manager.cc:1939] T c2632bc1ec2c4f3996e4d2105707cac0 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting consensus metadata
W20260927 21:02:17.632958 24372 connection.cc:441] server connection from 127.0.0.1:34202 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34202 (request call id 1) could send its response
I20260927 21:02:17.634016 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.620766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.16.190.252:41353)
0927 21:02:17.620802 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.620804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.624998 (+ 4194us) server_negotiation.cc:407] Connection header received
0927 21:02:17.625227 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.625229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.625246 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.625277 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.626390 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.627174 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.628216 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.628516 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.628559 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.630513 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.630514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.630515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.630523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.632560 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.633031 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.633033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.633033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.633091 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.633353 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.633354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.633354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.633490 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.633571 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.633825 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.633921 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:02:17.645586 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.633265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.16.190.252:41353)
0927 21:02:17.633316 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.633318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.637033 (+ 3715us) server_negotiation.cc:407] Connection header received
0927 21:02:17.637100 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.637102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.637114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.637171 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.638262 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.639067 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.640262 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.640574 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.640650 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.642231 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.642233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.642233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.642241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.644152 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.644585 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.644587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.644587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.644653 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.644902 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.644903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.644903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.645051 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.645151 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.645367 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.645458 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:02:17.646950 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:17.633722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.16.190.253:34651)
0927 21:02:17.633771 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:17.633773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:17.639453 (+ 5680us) server_negotiation.cc:407] Connection header received
0927 21:02:17.639520 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:17.639522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:17.639538 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:17.639601 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:17.640303 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.640934 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.641965 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:17.642292 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:17.642369 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:17.643891 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:17.643893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:17.643893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:17.643900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:17.645687 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.646167 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.646168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.646169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.646227 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:17.646413 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:17.646414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:17.646414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:17.646559 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:17.646637 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:17.646812 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 21:02:17.646887 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:02:18.616477 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:18.616159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:48627)
0927 21:02:18.616337 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:18.616414 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260927 21:02:18.616588 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:18.616206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:50913)
0927 21:02:18.616485 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:18.616548 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":53,"threads_started":1}
W20260927 21:02:18.616724 24207 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
W20260927 21:02:18.616909 24346 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:18.617697 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:18.617436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:57043)
0927 21:02:18.617584 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:18.617650 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260927 21:02:18.617892 24483 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.120394 24550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66b0334abc594ee6832dd56fe5e8d268)
I20260927 21:02:19.120556 24550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:19.120716 24551 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 66b0334abc594ee6832dd56fe5e8d268)
I20260927 21:02:19.120886 24550 leader_election.cc:288] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353)
I20260927 21:02:19.120846 24551 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:19.121389 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51490)
0927 21:02:19.121273 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.121332 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
W20260927 21:02:19.121527 23827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.121541 24551 leader_election.cc:288] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253:34651)
I20260927 21:02:19.121826 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51506)
0927 21:02:19.121734 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.121782 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":51,"threads_started":1}
W20260927 21:02:19.121944 24014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.122205 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51520)
0927 21:02:19.122149 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.122180 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260927 21:02:19.122290 23827 leader_election.cc:334] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.122530 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.122417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51526)
0927 21:02:19.122469 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.122503 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:02:19.122655 24014 leader_election.cc:334] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.124459 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:41353 (local address 127.0.0.1:58780)
0927 21:02:19.121248 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.121258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:19.121307 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:19.121686 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:19.121688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:19.121690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:19.121752 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:19.121754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.122818 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.122819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:19.123625 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:19.123627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.124242 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.124243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:19.124303 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:19.124308 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:19.124362 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:02:19.124408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":43,"threads_started":1}
I20260927 21:02:19.124536 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.16.190.252:41353)
0927 21:02:19.121338 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:19.121340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:19.121349 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:19.121375 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:19.121376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:19.121388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:19.121461 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:19.121913 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.122750 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.123708 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.124159 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.124214 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:19.124221 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:19.124397 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 21:02:19.124444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":42,"threads_started":1}
I20260927 21:02:19.124643 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34651 (local address 127.0.0.1:36898)
0927 21:02:19.121895 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.121905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:19.121957 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:19.122091 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:19.122092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:19.122094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:19.122141 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:19.122142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.123021 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.123022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:19.123813 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:19.123814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.124401 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.124402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:19.124467 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:19.124471 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:19.124536 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:02:19.124574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":43,"threads_started":1}
I20260927 21:02:19.124745 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.121698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.16.190.253:34651)
0927 21:02:19.121857 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:19.121859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:19.121996 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:02:19.122017 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:19.122018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:19.122028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:19.122090 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:19.122218 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.122946 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.123894 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:19.124288 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:19.124362 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:19.124369 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:19.124635 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 21:02:19.124686 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260927 21:02:19.124797 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50e1690dddf4d568459a7c3aa33abb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88b411d25b041eeb1430b20aa493be5" is_pre_election: true
I20260927 21:02:19.124948 24037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50e1690dddf4d568459a7c3aa33abb6 in term 1.
I20260927 21:02:19.124930 23851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a88b411d25b041eeb1430b20aa493be5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e50e1690dddf4d568459a7c3aa33abb6" is_pre_election: true
I20260927 21:02:19.125041 23851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88b411d25b041eeb1430b20aa493be5 in term 1.
I20260927 21:02:19.125200 24012 leader_election.cc:302] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a88b411d25b041eeb1430b20aa493be5, e50e1690dddf4d568459a7c3aa33abb6; no voters: 66b0334abc594ee6832dd56fe5e8d268
I20260927 21:02:19.125205 23828 leader_election.cc:302] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a88b411d25b041eeb1430b20aa493be5, e50e1690dddf4d568459a7c3aa33abb6; no voters: 66b0334abc594ee6832dd56fe5e8d268
I20260927 21:02:19.125300 24551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:02:19.125324 24550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 21:02:19.125344 24551 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66b0334abc594ee6832dd56fe5e8d268)
I20260927 21:02:19.125356 24550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 66b0334abc594ee6832dd56fe5e8d268)
I20260927 21:02:19.125365 24551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:02:19.125391 24550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:02:19.126052 24551 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:19.126070 24550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:19.126184 24551 leader_election.cc:288] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 election: Requested vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), e50e1690dddf4d568459a7c3aa33abb6 (127.16.190.253:34651)
I20260927 21:02:19.126205 24550 leader_election.cc:288] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 election: Requested vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353)
I20260927 21:02:19.126358 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.126265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51534)
0927 21:02:19.126303 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.126333 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 21:02:19.126441 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.126330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51536)
0927 21:02:19.126368 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.126393 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:02:19.126457 23851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a88b411d25b041eeb1430b20aa493be5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e50e1690dddf4d568459a7c3aa33abb6"
I20260927 21:02:19.126482 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50e1690dddf4d568459a7c3aa33abb6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88b411d25b041eeb1430b20aa493be5"
I20260927 21:02:19.126530 23851 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a88b411d25b041eeb1430b20aa493be5 in current term 2: Already voted for candidate e50e1690dddf4d568459a7c3aa33abb6 in this term.
I20260927 21:02:19.126557 24037 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e50e1690dddf4d568459a7c3aa33abb6 in current term 2: Already voted for candidate a88b411d25b041eeb1430b20aa493be5 in this term.
I20260927 21:02:19.126773 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.126633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51540)
0927 21:02:19.126726 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.126753 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 21:02:19.126812 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.126736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:51548)
0927 21:02:19.126766 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.126789 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 21:02:19.126835 23827 leader_election.cc:334] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.126871 23827 leader_election.cc:302] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e50e1690dddf4d568459a7c3aa33abb6; no voters: 66b0334abc594ee6832dd56fe5e8d268, a88b411d25b041eeb1430b20aa493be5
W20260927 21:02:19.126917 24014 leader_election.cc:334] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:19.126951 24014 leader_election.cc:302] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a88b411d25b041eeb1430b20aa493be5; no voters: 66b0334abc594ee6832dd56fe5e8d268, e50e1690dddf4d568459a7c3aa33abb6
I20260927 21:02:19.126950 24550 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 21:02:19.127028 24551 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 21:02:19.617571 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.617232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:48949)
0927 21:02:19.617424 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.617467 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
I20260927 21:02:19.617779 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.617116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:51777)
0927 21:02:19.617619 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.617661 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260927 21:02:19.618410 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:19.618186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:55823)
0927 21:02:19.618324 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:19.618364 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260927 21:02:20.618443 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.618075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:48417)
0927 21:02:20.618319 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.618382 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":63,"threads_started":1}
I20260927 21:02:20.618747 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.618391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:50431)
0927 21:02:20.618531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.618692 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260927 21:02:20.619031 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.618826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:45449)
0927 21:02:20.618933 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.618971 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260927 21:02:20.828197 24566 raft_consensus.cc:482] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:20.828305 24566 raft_consensus.cc:504] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:20.828611 24566 leader_election.cc:288] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353)
I20260927 21:02:20.828938 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50e1690dddf4d568459a7c3aa33abb6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88b411d25b041eeb1430b20aa493be5" is_pre_election: true
I20260927 21:02:20.829043 24037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e50e1690dddf4d568459a7c3aa33abb6 in term 2.
I20260927 21:02:20.829018 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.828784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55612)
0927 21:02:20.828942 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.828979 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260927 21:02:20.829253 23828 leader_election.cc:302] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88b411d25b041eeb1430b20aa493be5, e50e1690dddf4d568459a7c3aa33abb6; no voters:
I20260927 21:02:20.829349 24566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 21:02:20.829402 24566 raft_consensus.cc:482] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:20.829427 24566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:02:20.829524 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.829394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55614)
0927 21:02:20.829458 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.829497 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 21:02:20.829624 23827 leader_election.cc:334] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:20.830259 24566 raft_consensus.cc:504] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:20.830420 24566 leader_election.cc:288] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 election: Requested vote from peers 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891), a88b411d25b041eeb1430b20aa493be5 (127.16.190.252:41353)
I20260927 21:02:20.830588 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.830488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55628)
0927 21:02:20.830527 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.830564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 21:02:20.830612 24037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e50e1690dddf4d568459a7c3aa33abb6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a88b411d25b041eeb1430b20aa493be5"
I20260927 21:02:20.830749 24037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:02:20.831041 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.830881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55632)
0927 21:02:20.830990 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.831020 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260927 21:02:20.831121 23827 leader_election.cc:334] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
I20260927 21:02:20.831604 24037 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e50e1690dddf4d568459a7c3aa33abb6 in term 3.
I20260927 21:02:20.831774 23828 leader_election.cc:302] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a88b411d25b041eeb1430b20aa493be5, e50e1690dddf4d568459a7c3aa33abb6; no voters: 66b0334abc594ee6832dd56fe5e8d268
I20260927 21:02:20.831915 24566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:02:20.832142 24566 raft_consensus.cc:686] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [term 3 LEADER]: Becoming Leader. State: Replica: e50e1690dddf4d568459a7c3aa33abb6, State: Running, Role: LEADER
I20260927 21:02:20.832232 24566 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } }
I20260927 21:02:20.832626 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1690dddf4d568459a7c3aa33abb6. Latest consensus state: current_term: 3 leader_uuid: "e50e1690dddf4d568459a7c3aa33abb6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:20.832728 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:20.832907 24571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:20.833273 24571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:20.833324 24571 catalog_manager.cc:1259] Loaded cluster ID: e5d5347804c04598b3ce8fd71157fd33
I20260927 21:02:20.833339 24571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:20.833488 24571 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:20.833544 24571 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Loaded TSK: 0
I20260927 21:02:20.833631 24571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:20.886096 23840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:20.887816 23840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:20.889014 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.888883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55650)
0927 21:02:20.888940 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.888981 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:02:20.889060 24037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [term 3 FOLLOWER]: Refusing update from remote peer e50e1690dddf4d568459a7c3aa33abb6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 21:02:20.889371 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.889298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.0.0.1:55664)
0927 21:02:20.889327 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.889354 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 21:02:20.889304 24569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 21:02:20.889468 23827 consensus_peers.cc:487] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 -> Peer 66b0334abc594ee6832dd56fe5e8d268 (127.16.190.254:37891): Couldn't send request to peer 66b0334abc594ee6832dd56fe5e8d268. Status: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 21:02:20.889995 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50e1690dddf4d568459a7c3aa33abb6. Latest consensus state: current_term: 3 leader_uuid: "e50e1690dddf4d568459a7c3aa33abb6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:20.890108 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:02:20.890457 24572 mvcc.cc:205] Tried to move back new op lower bound from 7334063885878829056 to 7334063885649584128. Current Snapshot: MvccSnapshot[applied={T|T < 7334063885878829056}]
I20260927 21:02:20.890573 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e50e1690dddf4d568459a7c3aa33abb6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:20.890666 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:20.891007 24573 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e50e1690dddf4d568459a7c3aa33abb6" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "66b0334abc594ee6832dd56fe5e8d268" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37891 } } peers { permanent_uuid: "e50e1690dddf4d568459a7c3aa33abb6" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34651 } attrs { promote: false } } peers { permanent_uuid: "a88b411d25b041eeb1430b20aa493be5" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 41353 } attrs { promote: false } } }
I20260927 21:02:20.891100 24573 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b411d25b041eeb1430b20aa493be5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:02:20.895694 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:43739 (local address 127.0.0.1:40898)
0927 21:02:20.892518 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.892529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:20.892582 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:20.892897 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:20.892898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:20.892900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:20.892959 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.892960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.893728 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.893729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.894798 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.894800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.895458 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.895459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.895519 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.895523 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:20.895577 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:02:20.895621 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":1}
I20260927 21:02:20.895704 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.16.190.193:43739)
0927 21:02:20.892727 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.892729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.892735 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:02:20.892761 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.892762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.892774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.892893 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:20.893052 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.893640 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.894895 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.895377 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.895464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.895472 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.895593 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.895635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260927 21:02:20.895756 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.16.190.195:35689)
0927 21:02:20.892599 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.892601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.892647 (+ 46us) server_negotiation.cc:407] Connection header received
0927 21:02:20.892736 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.892737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.892750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.892862 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:20.893076 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.893854 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.894838 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.895393 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.895441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.895449 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.895643 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.895678 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
I20260927 21:02:20.895984 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:35689 (local address 127.0.0.1:34166)
0927 21:02:20.892556 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.892603 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:20.892656 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:20.892920 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:20.892921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:20.892924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:20.892977 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.892979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.893953 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.893954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.894734 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.894736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.895506 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.895507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.895566 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.895570 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:20.895854 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 21:02:20.895909 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260927 21:02:20.896225 24417 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 (DEFAULT_TABLE table=test-table [id=cf6504243b49480abb7f46f968d2051a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:20.896261 24141 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 (DEFAULT_TABLE table=test-table [id=cf6504243b49480abb7f46f968d2051a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:20.896390 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef54f94e33f42b3b72753943f3d6291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:20.896414 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef54f94e33f42b3b72753943f3d6291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:20.897614 24583 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Bootstrap starting.
I20260927 21:02:20.897986 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.16.190.194:41531)
0927 21:02:20.892772 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.892774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.892782 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:20.892973 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.892974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.892986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.893052 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:20.893224 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.893946 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.895170 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.895939 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.896012 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.896090 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.897863 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.897916 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":59,"threads_started":1}
I20260927 21:02:20.897989 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.892407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:41531 (local address 127.0.0.1:58630)
0927 21:02:20.892674 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.892683 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:20.892719 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:20.893098 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:20.893098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:20.893100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:20.893141 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.893142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.894024 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.894025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.895091 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.895093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.896015 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.896016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.897780 (+ 1764us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.897785 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:20.897845 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:02:20.897886 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260927 21:02:20.898005 24583 tablet_bootstrap.cc:654] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:20.898232 24584 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: Bootstrap starting.
I20260927 21:02:20.898248 24279 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 (DEFAULT_TABLE table=test-table [id=cf6504243b49480abb7f46f968d2051a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:20.898365 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef54f94e33f42b3b72753943f3d6291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:20.898610 24584 tablet_bootstrap.cc:654] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:20.898660 24583 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: No bootstrap required, opened a new log
I20260927 21:02:20.898700 24583 ts_tablet_manager.cc:1397] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:20.898861 24583 raft_consensus.cc:348] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.898962 24583 raft_consensus.cc:374] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:20.898990 24583 raft_consensus.cc:729] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2213654f514bfbb47eb3b9b0ff9efa, State: Initialized, Role: FOLLOWER
I20260927 21:02:20.899039 24583 consensus_queue.cc:260] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.899235 24583 ts_tablet_manager.cc:1428] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:20.899461 24584 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: No bootstrap required, opened a new log
I20260927 21:02:20.899549 24584 ts_tablet_manager.cc:1397] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:20.899686 24584 raft_consensus.cc:348] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.899755 24584 raft_consensus.cc:374] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:20.899770 24584 raft_consensus.cc:729] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f98770f6825405a98b88d0780517753, State: Initialized, Role: FOLLOWER
I20260927 21:02:20.899837 24584 consensus_queue.cc:260] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.900012 24584 ts_tablet_manager.cc:1428] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 21:02:20.900022 24486 tablet.cc:2369] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:20.900075 24588 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Bootstrap starting.
I20260927 21:02:20.900404 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.900271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:58257)
0927 21:02:20.900340 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.900376 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:02:20.900496 24588 tablet_bootstrap.cc:654] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:20.900883 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.900614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:60007)
0927 21:02:20.900778 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.900859 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260927 21:02:20.900979 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.900324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:42711)
0927 21:02:20.900918 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.900951 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575}
I20260927 21:02:20.901229 24588 tablet_bootstrap.cc:492] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: No bootstrap required, opened a new log
I20260927 21:02:20.901280 24588 ts_tablet_manager.cc:1397] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:20.901422 24588 raft_consensus.cc:348] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.901520 24588 raft_consensus.cc:374] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:20.901553 24588 raft_consensus.cc:729] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693503666d254f348932e12a588a1448, State: Initialized, Role: FOLLOWER
I20260927 21:02:20.901634 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.901214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:35321)
0927 21:02:20.901578 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.901608 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260927 21:02:20.901628 24588 consensus_queue.cc:260] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.901803 24588 ts_tablet_manager.cc:1428] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:20.902057 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.901970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:50899)
0927 21:02:20.902011 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.902034 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 21:02:20.902281 24348 heartbeater.cc:503] Master 127.16.190.253:34651 was elected leader, sending a full tablet report...
I20260927 21:02:20.902326 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.902251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:36923)
0927 21:02:20.902280 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.902305 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 21:02:20.963230 24587 raft_consensus.cc:482] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:20.963339 24587 raft_consensus.cc:504] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.963810 24587 leader_election.cc:288] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739), 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:20.967859 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.964028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:41531 (local address 127.16.190.195:43911)
0927 21:02:20.964315 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.964324 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:20.964393 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:20.964555 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:20.964556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:20.964559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:20.964621 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.964623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.965574 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.965575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.966803 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.966806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.967591 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.967592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.967656 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.967661 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:20.967721 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:02:20.967763 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":83,"thread_start_us":128,"threads_started":1}
I20260927 21:02:20.967873 24580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.964129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:43911 (local address 127.16.190.194:41531)
0927 21:02:20.964255 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.964257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.964392 (+ 135us) server_negotiation.cc:407] Connection header received
0927 21:02:20.964454 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.964456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.964469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.964537 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:20.964716 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.965493 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.966941 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.967523 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.967586 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.967597 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.967732 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.967775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 21:02:20.968111 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef54f94e33f42b3b72753943f3d6291" candidate_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693503666d254f348932e12a588a1448" is_pre_election: true
I20260927 21:02:20.968230 24299 raft_consensus.cc:2454] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2213654f514bfbb47eb3b9b0ff9efa in term 0.
I20260927 21:02:20.968380 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.964028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:43739 (local address 127.16.190.195:60217)
0927 21:02:20.964114 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.964130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:20.964212 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:20.965030 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:20.965030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:20.965033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:20.965070 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.965071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.966013 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.966014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.967131 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:20.967134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.968123 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.968124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:20.968213 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.968217 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:20.968274 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 21:02:20.968317 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 21:02:20.968586 24369 leader_election.cc:302] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2213654f514bfbb47eb3b9b0ff9efa, 693503666d254f348932e12a588a1448; no voters:
I20260927 21:02:20.968675 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.964203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:60217 (local address 127.16.190.193:43739)
0927 21:02:20.964283 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.964286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.964298 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:20.964935 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.964937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.964947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.965000 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:20.965141 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.965884 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.967234 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.967892 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.967962 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.967973 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.968556 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.968610 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:02:20.968740 24587 raft_consensus.cc:2781] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:20.968786 24587 raft_consensus.cc:482] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:20.968837 24587 raft_consensus.cc:3037] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:20.969000 24161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef54f94e33f42b3b72753943f3d6291" candidate_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f98770f6825405a98b88d0780517753" is_pre_election: true
I20260927 21:02:20.969144 24161 raft_consensus.cc:2454] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2213654f514bfbb47eb3b9b0ff9efa in term 0.
I20260927 21:02:20.969717 24587 raft_consensus.cc:504] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.969864 24587 leader_election.cc:288] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [CANDIDATE]: Term 1 election: Requested vote from peers 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739), 693503666d254f348932e12a588a1448 (127.16.190.194:41531)
I20260927 21:02:20.970044 24161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef54f94e33f42b3b72753943f3d6291" candidate_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f98770f6825405a98b88d0780517753"
I20260927 21:02:20.970044 24299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef54f94e33f42b3b72753943f3d6291" candidate_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693503666d254f348932e12a588a1448"
I20260927 21:02:20.970119 24161 raft_consensus.cc:3037] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:20.970120 24299 raft_consensus.cc:3037] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:20.970912 24161 raft_consensus.cc:2454] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2213654f514bfbb47eb3b9b0ff9efa in term 1.
I20260927 21:02:20.970912 24299 raft_consensus.cc:2454] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2213654f514bfbb47eb3b9b0ff9efa in term 1.
I20260927 21:02:20.971076 24372 leader_election.cc:302] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2213654f514bfbb47eb3b9b0ff9efa, 9f98770f6825405a98b88d0780517753; no voters:
I20260927 21:02:20.971169 24587 raft_consensus.cc:2781] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:20.971335 24587 raft_consensus.cc:686] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 LEADER]: Becoming Leader. State: Replica: 3e2213654f514bfbb47eb3b9b0ff9efa, State: Running, Role: LEADER
I20260927 21:02:20.971429 24485 heartbeater.cc:503] Master 127.16.190.253:34651 was elected leader, sending a full tablet report...
I20260927 21:02:20.971419 24587 consensus_queue.cc:237] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } }
I20260927 21:02:20.971693 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.971576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:42441)
0927 21:02:20.971629 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.971660 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:02:20.972167 23841 catalog_manager.cc:5495] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195). New cstate: current_term: 1 leader_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2213654f514bfbb47eb3b9b0ff9efa" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 35689 } health_report { overall_health: HEALTHY } } }
I20260927 21:02:20.990751 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.987897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.16.190.195:35689)
0927 21:02:20.987983 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:20.987985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:20.987994 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:20.988034 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:20.988035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:20.988055 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:20.988145 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:20.988363 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.988980 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.989797 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:20.990044 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:20.990118 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:20.990330 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:20.990443 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:20.990567 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:20.990584 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 21:02:20.990639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:02:20.991984 24299 raft_consensus.cc:1261] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Refusing update from remote peer 3e2213654f514bfbb47eb3b9b0ff9efa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:20.992079 24161 raft_consensus.cc:1261] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Refusing update from remote peer 3e2213654f514bfbb47eb3b9b0ff9efa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:20.992316 24593 consensus_queue.cc:1035] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "693503666d254f348932e12a588a1448" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:20.992442 24587 consensus_queue.cc:1035] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f98770f6825405a98b88d0780517753" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:20.993366 24209 heartbeater.cc:503] Master 127.16.190.253:34651 was elected leader, sending a full tablet report...
I20260927 21:02:20.993525 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.993411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:45091)
0927 21:02:20.993456 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.993490 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 21:02:20.993634 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.993353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:55397)
0927 21:02:20.993560 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.993595 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260927 21:02:20.994882 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.994775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.195:60125)
0927 21:02:20.994818 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.994846 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 21:02:20.995107 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.994941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.193:58039)
0927 21:02:20.995047 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.995074 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 21:02:20.995193 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:20.995087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37891 (local address 127.16.190.194:49105)
0927 21:02:20.995137 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:20.995164 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:02:20.995900 24597 mvcc.cc:205] Tried to move back new op lower bound from 7334063886300991488 to 7334063886219632640. Current Snapshot: MvccSnapshot[applied={T|T < 7334063886300991488 or (T in {7334063886300991488})}]
I20260927 21:02:20.997977 23840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:20.998081 23840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:20.999436 23840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e50e1690dddf4d568459a7c3aa33abb6: Sending DeleteTablet for 3 replicas of tablet 2ef54f94e33f42b3b72753943f3d6291
I20260927 21:02:20.999797 24279 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:20 UTC) from {username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:58630
I20260927 21:02:20.999892 24417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:20 UTC) from {username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:34166
I20260927 21:02:21.000072 24141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef54f94e33f42b3b72753943f3d6291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:20 UTC) from {username='oryx', principal='oryx/127.16.190.253@KRBTEST.COM'} at 127.0.0.1:40898
I20260927 21:02:21.000131 24607 tablet_replica.cc:330] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: stopping tablet replica
I20260927 21:02:21.000190 24607 raft_consensus.cc:2229] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:21.000296 24607 raft_consensus.cc:2258] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:21.000324 24608 tablet_replica.cc:330] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753: stopping tablet replica
I20260927 21:02:21.000566 24608 raft_consensus.cc:2229] T 2ef54f94e33f42b3b72753943f3d6291 P 9f98770f6825405a98b88d0780517753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:21.000649 24606 tablet_replica.cc:330] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: stopping tablet replica
I20260927 21:02:21.000691 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23989
I20260927 21:02:21.000710 24606 raft_consensus.cc:2229] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:21.000763 24606 raft_consensus.cc:2258] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:21.007697 24606 ts_tablet_manager.cc:1905] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:21.008154 24607 ts_tablet_manager.cc:1905] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:21.010265 24607 ts_tablet_manager.cc:1918] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 21:02:21.010509 24578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:21.010360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:43739 (local address 127.0.0.1:40904)
0927 21:02:21.010414 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:21.010441 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:43739: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:02:21.010639 23828 catalog_manager.cc:4584] TS 9f98770f6825405a98b88d0780517753 (127.16.190.193:43739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ef54f94e33f42b3b72753943f3d6291: Network error: Client connection negotiation failed: client connection to 127.16.190.193:43739: connect: Connection reset by peer (error 104)
I20260927 21:02:21.010748 24607 log.cc:1196] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/2ef54f94e33f42b3b72753943f3d6291
I20260927 21:02:21.010751 24606 ts_tablet_manager.cc:1918] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:21.010823 24606 log.cc:1196] T 2ef54f94e33f42b3b72753943f3d6291 P 693503666d254f348932e12a588a1448: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/2ef54f94e33f42b3b72753943f3d6291
I20260927 21:02:21.010816 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 24215
I20260927 21:02:21.011119 24607 ts_tablet_manager.cc:1939] T 2ef54f94e33f42b3b72753943f3d6291 P 3e2213654f514bfbb47eb3b9b0ff9efa: Deleting consensus metadata
I20260927 21:02:21.011837 23826 catalog_manager.cc:4841] TS 3e2213654f514bfbb47eb3b9b0ff9efa (127.16.190.195:35689): tablet 2ef54f94e33f42b3b72753943f3d6291 (table test-table [id=cf6504243b49480abb7f46f968d2051a]) successfully deleted
I20260927 21:02:21.016289 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 24353
I20260927 21:02:21.016322 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:21.016174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:41531 (local address 127.0.0.1:58644)
0927 21:02:21.016246 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:21.016282 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.194:41531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:02:21.020692 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23707
I20260927 21:02:21.025575 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 23895
2026-09-27T21:02:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6498 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:21 dist-test-slave-6c7s krb5kdc[24611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:21 dist-test-slave-6c7s krb5kdc[24611](info): set up 2 sockets
Sep 27 21:02:21 dist-test-slave-6c7s krb5kdc[24611](info): commencing operation
krb5kdc: starting...
W20260927 21:02:22.075083 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab.
Entry for principal oryx/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab.
I20260927 21:02:22.199858 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:32967
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab
--principal=oryx/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:22.281669 24624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:22.281898 24624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:22.281924 24624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:22.283030 24624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:22.283085 24624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 21:02:22.283102 24624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:22.283115 24624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:22.283128 24624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:22.283140 24624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:22.284762 24624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:32967
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.254.keytab
--principal=oryx/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:22.285017 24624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:22.285204 24624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:22.290372 24630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.290366 24629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.290388 24632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:22.290706 24624 server_base.cc:1034] running on GCE node
I20260927 21:02:22.290961 24624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:22.291289 24624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:22.292470 24624 hybrid_clock.cc:648] HybridClock initialized: now 1790542942292457 us; error 44 us; skew 500 ppm
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:22.295210 24624 init.cc:377] Logged in from keytab as oryx/127.16.190.254@KRBTEST.COM (short username oryx)
I20260927 21:02:22.295629 24624 webserver.cc:460] Webserver started at http://127.16.190.254:40017/ using document root <none> and password file <none>
I20260927 21:02:22.295888 24624 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:22.295936 24624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:22.296072 24624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:22.297266 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "c50513fd02c9480ca9b366f2714b9428"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "24b3ca39356931cfa312c0eab0d6f8f4"
server_key_iv: "3c2d0fe8185d59aeb8b5c8b4d43cb970"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.297657 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "c50513fd02c9480ca9b366f2714b9428"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "24b3ca39356931cfa312c0eab0d6f8f4"
server_key_iv: "3c2d0fe8185d59aeb8b5c8b4d43cb970"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.299100 24624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:22.299962 24639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.300190 24624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:22.300282 24624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "c50513fd02c9480ca9b366f2714b9428"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "24b3ca39356931cfa312c0eab0d6f8f4"
server_key_iv: "3c2d0fe8185d59aeb8b5c8b4d43cb970"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.300354 24624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:22.316767 24624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:22.319106 24624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:22.319231 24624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:22.324131 24624 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:32967
I20260927 21:02:22.324226 24702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:32967 every 8 connection(s)
I20260927 21:02:22.324573 24624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:22.324954 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 24624
I20260927 21:02:22.325066 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:22.325277 17147 external_mini_cluster.cc:1428] Setting key 0e99e0131f431be58938eac09afcd2de
I20260927 21:02:22.325423 24703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.328694 24703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Bootstrap starting.
I20260927 21:02:22.329454 24703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:22.329942 24703 log.cc:824] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:22.330696 24703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: No bootstrap required, opened a new log
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:22.332736 24703 raft_consensus.cc:348] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } }
I20260927 21:02:22.332985 24703 raft_consensus.cc:374] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:22.333076 24703 raft_consensus.cc:729] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c50513fd02c9480ca9b366f2714b9428, State: Initialized, Role: FOLLOWER
I20260927 21:02:22.333241 24703 consensus_queue.cc:260] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } }
I20260927 21:02:22.333348 24703 raft_consensus.cc:388] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:22.333384 24703 raft_consensus.cc:482] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:22.333433 24703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:22.334203 24703 raft_consensus.cc:504] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } }
I20260927 21:02:22.334321 24703 leader_election.cc:302] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c50513fd02c9480ca9b366f2714b9428; no voters:
I20260927 21:02:22.334491 24703 leader_election.cc:288] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:22.334587 24708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:22.334765 24708 raft_consensus.cc:686] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [term 1 LEADER]: Becoming Leader. State: Replica: c50513fd02c9480ca9b366f2714b9428, State: Running, Role: LEADER
I20260927 21:02:22.334859 24708 consensus_queue.cc:237] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } }
I20260927 21:02:22.334894 24703 sys_catalog.cc:564] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:22.335408 24708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c50513fd02c9480ca9b366f2714b9428. Latest consensus state: current_term: 1 leader_uuid: "c50513fd02c9480ca9b366f2714b9428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } } }
I20260927 21:02:22.335485 24708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:22.335384 24709 sys_catalog.cc:455] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c50513fd02c9480ca9b366f2714b9428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c50513fd02c9480ca9b366f2714b9428" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 32967 } } }
I20260927 21:02:22.335578 24709 sys_catalog.cc:458] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:22.337136 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.325846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.16.190.254:32967)
0927 21:02:22.326028 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.326037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.326090 (+ 53us) server_negotiation.cc:407] Connection header received
0927 21:02:22.326521 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.326528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.326576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.326691 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.327521 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.328414 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.329525 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.329971 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.330125 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.332608 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.332613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.332614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.332629 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.334966 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.335481 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.335484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.335484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.335609 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.335822 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.335823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.335823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.336189 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.336342 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.336748 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.336923 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
W20260927 21:02:22.337320 24723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:02:22.337368 24723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:02:22.337489 24724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:22.337654 24724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:22.339257 24724 catalog_manager.cc:1347] Generated new cluster ID: a2f927d1f59446839c48f8a08c933398
I20260927 21:02:22.339316 24724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:22.344643 24724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:22.345259 24724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:22.355479 24724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Generated new TSK 0
I20260927 21:02:22.355707 24724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
I20260927 21:02:22.390028 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:32967
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:22.473160 24729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:22.473379 24729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:22.473407 24729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:22.474754 24729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:22.474802 24729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:22.474823 24729 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:02:22.474841 24729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:22.476373 24729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:22.476626 24729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:22.476903 24729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:22.481864 24734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.481895 24737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.481884 24735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:22.482123 24729 server_base.cc:1034] running on GCE node
I20260927 21:02:22.482347 24729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:22.482556 24729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:22.483695 24729 hybrid_clock.cc:648] HybridClock initialized: now 1790542942483679 us; error 28 us; skew 500 ppm
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:22.486410 24729 init.cc:377] Logged in from keytab as oryx/127.16.190.193@KRBTEST.COM (short username oryx)
I20260927 21:02:22.486892 24729 webserver.cc:460] Webserver started at http://127.16.190.193:43681/ using document root <none> and password file <none>
I20260927 21:02:22.487079 24729 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:22.487119 24729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:22.487215 24729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:22.488245 24729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "5bf66e60dc3a4ff1877b17a34086a907"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "2d4756c825d78b9310d1c65085d5a627"
server_key_iv: "d03a9ac5d6f3f918572abe78c6e4bf8b"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.488551 24729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf66e60dc3a4ff1877b17a34086a907"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "2d4756c825d78b9310d1c65085d5a627"
server_key_iv: "d03a9ac5d6f3f918572abe78c6e4bf8b"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.489912 24729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:22.490628 24744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.490824 24729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:22.490932 24729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "5bf66e60dc3a4ff1877b17a34086a907"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "2d4756c825d78b9310d1c65085d5a627"
server_key_iv: "d03a9ac5d6f3f918572abe78c6e4bf8b"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.491014 24729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:22.498361 24729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:22.500504 24729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:22.500665 24729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:22.501222 24729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:22.501262 24729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.501284 24729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:22.501300 24729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.508018 24729 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:44073
I20260927 21:02:22.508241 24860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:44073 every 8 connection(s)
I20260927 21:02:22.508476 24729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:22.513453 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.501814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:47525 (local address 127.16.190.254:32967)
0927 21:02:22.501863 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.501865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.502091 (+ 226us) server_negotiation.cc:407] Connection header received
0927 21:02:22.502687 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.502689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.502704 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.502787 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.503801 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.504729 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.506366 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.506758 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.506843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.509655 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.509656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.509657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.509664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.511845 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.512299 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.512301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.512302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.512360 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.512653 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.512654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.512655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.512851 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.512919 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.513269 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.513356 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:02:22.513667 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.501810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.193:47525)
0927 21:02:22.501962 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.501999 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.502522 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.502887 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.502906 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.503365 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.503679 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.503685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.504854 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.504855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.506268 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.506271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.506852 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.506853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.507009 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.507690 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.507698 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.509525 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.511953 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.511975 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.511977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.512219 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.512448 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.512449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.512449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.512562 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.512916 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.512924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.513112 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.513236 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.513405 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260927 21:02:22.514189 24861 heartbeater.cc:347] Connected to a master server at 127.16.190.254:32967
I20260927 21:02:22.514282 24861 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:22.514479 24861 heartbeater.cc:511] Master 127.16.190.254:32967 requested a full tablet report, sending...
I20260927 21:02:22.515033 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 24729
I20260927 21:02:22.515146 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:22.515144 24655 ts_manager.cc:194] Registered new tserver with Master: 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073)
I20260927 21:02:22.515337 17147 external_mini_cluster.cc:1428] Setting key 076d7ce20ffda1b93afbec7aafff8c0d
I20260927 21:02:22.516075 24655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:47525
I20260927 21:02:22.521971 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.515712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:43755 (local address 127.16.190.254:32967)
0927 21:02:22.515769 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.515771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.516039 (+ 268us) server_negotiation.cc:407] Connection header received
0927 21:02:22.516102 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.516103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.516113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.516164 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.516732 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.517279 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.518050 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.518258 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.518336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.519306 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.519307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.519308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.519314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.520752 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.521167 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.521168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.521169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.521225 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.521437 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.521438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.521438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.521554 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.521631 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.521793 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.521874 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:02:22.521984 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.515683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.193:43755)
0927 21:02:22.515981 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.515994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.516054 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.516212 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.516213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.516511 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.516612 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.516614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.517380 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.517382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.517944 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.517947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.518345 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.518346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.518412 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.518899 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.518904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.519196 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.520895 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.520896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.520897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.521074 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.521308 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.521309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.521309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.521374 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.521623 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.521624 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.521681 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.521767 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.521876 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
I20260927 21:02:22.543251 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:32967
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:22.626531 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:22.626742 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:22.626767 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:22.627944 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:22.627990 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:22.628010 24870 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:02:22.628069 24870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:22.629706 24870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:22.629954 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:22.630215 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:22.635574 24870 server_base.cc:1034] running on GCE node
W20260927 21:02:22.635462 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.635473 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.635463 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:22.636152 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:22.636428 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:22.637622 24870 hybrid_clock.cc:648] HybridClock initialized: now 1790542942637613 us; error 35 us; skew 500 ppm
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:22.640340 24870 init.cc:377] Logged in from keytab as oryx/127.16.190.194@KRBTEST.COM (short username oryx)
I20260927 21:02:22.640894 24870 webserver.cc:460] Webserver started at http://127.16.190.194:35673/ using document root <none> and password file <none>
I20260927 21:02:22.641117 24870 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:22.641160 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:22.641286 24870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:22.642336 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "909cf0089dba42999d75678728cf6cba"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "6177c905d5293265263de5133c056718"
server_key_iv: "208611f9834b2004598155d79e34dd7a"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.642650 24870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "909cf0089dba42999d75678728cf6cba"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "6177c905d5293265263de5133c056718"
server_key_iv: "208611f9834b2004598155d79e34dd7a"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.643960 24870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:22.644701 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.644918 24870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:22.645011 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "909cf0089dba42999d75678728cf6cba"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "6177c905d5293265263de5133c056718"
server_key_iv: "208611f9834b2004598155d79e34dd7a"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.645073 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:22.657968 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:22.660416 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:22.660534 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:22.661116 24870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:22.661187 24870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.661211 24870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:22.661223 24870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.668164 24870 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44659
I20260927 21:02:22.668223 25000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44659 every 8 connection(s)
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:22.668548 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:22.671876 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.661597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:36819 (local address 127.16.190.254:32967)
0927 21:02:22.661685 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.661687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.661855 (+ 168us) server_negotiation.cc:407] Connection header received
0927 21:02:22.662376 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.662378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.662392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.662477 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.663427 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.664290 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.665974 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.666297 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.666384 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.669133 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.669135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.669135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.669145 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.670431 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.670919 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.670920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.670920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.671002 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.671283 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.671284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.671285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.671406 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.671461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.671723 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.671806 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:02:22.671978 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.661571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.194:36819)
0927 21:02:22.661725 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.661766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.662274 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.662538 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.662541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.662976 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.663310 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.663317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.664401 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.664402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.665865 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.665870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.666396 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.666396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.666510 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.667148 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.667160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.669006 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.670539 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.670549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.670551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.670817 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.671098 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.671099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.671099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.671215 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.671448 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.671450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.671572 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.671663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.671803 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260927 21:02:22.672725 25001 heartbeater.cc:347] Connected to a master server at 127.16.190.254:32967
I20260927 21:02:22.672856 25001 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:22.673049 25001 heartbeater.cc:511] Master 127.16.190.254:32967 requested a full tablet report, sending...
I20260927 21:02:22.673485 24655 ts_manager.cc:194] Registered new tserver with Master: 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659)
I20260927 21:02:22.673923 24655 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:36819
I20260927 21:02:22.678239 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 24870
I20260927 21:02:22.678365 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:22.678614 17147 external_mini_cluster.cc:1428] Setting key 4b5de32fff03184f0c17cf39162f4d32
I20260927 21:02:22.679927 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.673909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.194:54717)
0927 21:02:22.674073 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.674086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.674135 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.674254 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.674255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.674446 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.674503 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.674504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.675188 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.675189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.675719 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.675720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.676102 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.676103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.676171 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.676652 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.676657 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.677060 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.678699 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.678700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.678701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.678982 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.679177 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.679178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.679178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.679250 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.679653 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.679653 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.679698 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.679756 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.679846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260927 21:02:22.680133 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.674000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:54717 (local address 127.16.190.254:32967)
0927 21:02:22.674110 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.674112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.674120 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:22.674184 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.674185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.674195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.674237 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.674592 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.675087 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.675808 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.676021 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.676105 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.677168 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.677170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.677170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.677177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.678593 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.679062 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.679062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.679063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.679113 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.679344 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.679345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.679345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.679573 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.679657 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.679806 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.679891 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for oryx/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
I20260927 21:02:22.704349 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:32967
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:22.789281 25010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:22.789515 25010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:22.789542 25010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:22.790772 25010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:22.790817 25010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:22.790839 25010 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 21:02:22.790858 25010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:22.792400 25010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.190.254:32967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:22.792614 25010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:22.792842 25010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:22.798280 25016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:22.798349 25015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:22.798368 25010 server_base.cc:1034] running on GCE node
W20260927 21:02:22.798280 25018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:22.798877 25010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:22.799170 25010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:22.800323 25010 hybrid_clock.cc:648] HybridClock initialized: now 1790542942800300 us; error 33 us; skew 500 ppm
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:22.803205 25010 init.cc:377] Logged in from keytab as oryx/127.16.190.195@KRBTEST.COM (short username oryx)
I20260927 21:02:22.803803 25010 webserver.cc:460] Webserver started at http://127.16.190.195:34585/ using document root <none> and password file <none>
I20260927 21:02:22.804111 25010 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:22.804160 25010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:22.804339 25010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:22.805610 25010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "4f3ac08225bb43331cd6f8de5b2f08c8"
server_key_iv: "2fe337a6ade8828e5d5c3866cb2a138e"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.806022 25010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "4f3ac08225bb43331cd6f8de5b2f08c8"
server_key_iv: "2fe337a6ade8828e5d5c3866cb2a138e"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.807441 25010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:22.808262 25025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.808473 25010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:22.808560 25010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
format_stamp: "Formatted at 2026-09-27 21:02:22 on dist-test-slave-6c7s"
server_key: "4f3ac08225bb43331cd6f8de5b2f08c8"
server_key_iv: "2fe337a6ade8828e5d5c3866cb2a138e"
server_key_version: "encryptionkey@0"
I20260927 21:02:22.808615 25010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:22.829849 25010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:22.832052 25010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:22.832181 25010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:22.832660 25010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:22.832702 25010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.832728 25010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:22.832741 25010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:22.840127 25010 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:46213
I20260927 21:02:22.840209 25140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:46213 every 8 connection(s)
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:22.840580 25010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:22.844079 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.833202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:59781 (local address 127.16.190.254:32967)
0927 21:02:22.833275 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.833279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.833434 (+ 155us) server_negotiation.cc:407] Connection header received
0927 21:02:22.833940 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.833941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.833956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.834039 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.834979 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.835895 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.837572 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.837984 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.838075 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.841026 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.841028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.841028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.841037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.842576 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.843031 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.843033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.843033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.843101 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.843387 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.843388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.843388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.843518 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.843592 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.843899 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.843985 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:02:22.844269 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.833157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.195:59781)
0927 21:02:22.833306 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.833347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.833846 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.834141 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.834145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.834560 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.834873 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.834882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.836003 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.836004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.837450 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.837453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.838090 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.838091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.838245 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.838876 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.838888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.840886 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.842694 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.842701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.842703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.842936 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.843189 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.843190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.843191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.843314 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.843597 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.843603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.843756 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.843862 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.844031 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260927 21:02:22.844772 25141 heartbeater.cc:347] Connected to a master server at 127.16.190.254:32967
I20260927 21:02:22.844895 25141 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:22.845079 25141 heartbeater.cc:511] Master 127.16.190.254:32967 requested a full tablet report, sending...
I20260927 21:02:22.845496 24656 ts_manager.cc:194] Registered new tserver with Master: bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:22.845899 24656 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:59781
I20260927 21:02:22.849792 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25010
I20260927 21:02:22.849920 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:22.850160 17147 external_mini_cluster.cc:1428] Setting key 6510eaa80f91691936fcd2f4710522e2
I20260927 21:02:22.851168 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:02:22.853053 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.846407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:53443 (local address 127.16.190.254:32967)
0927 21:02:22.846445 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.846447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.846497 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:02:22.846579 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.846580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.846588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.846642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.847007 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.847475 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.848557 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.848764 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.848917 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.849901 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.849903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.849903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.849912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.851580 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.852117 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.852118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.852119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.852189 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.852444 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.852445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.852445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.852566 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.852630 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.852891 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.852974 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:02:22.853097 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.846294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.16.190.195:53443)
0927 21:02:22.846439 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.846449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.846505 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.846670 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.846671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.846896 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:22.846947 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.846949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.847591 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.847592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.848447 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.848450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.848925 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.848926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.848983 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.849493 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:22.849497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:22.849798 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:22.851723 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.851724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.851725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.851980 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.852279 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:22.852280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:22.852280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:22.852356 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:22.852647 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:22.852648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:22.852702 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.852777 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.852936 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:02:22.874882 24706 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:22.868596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.16.190.254:32967)
0927 21:02:22.868672 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.868673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.868683 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:22.868729 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.868730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.868741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.868793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.869458 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.869958 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.870817 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.871016 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.871082 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.874772 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59960: BlockingRecv error: recv got EOF from 127.0.0.1:59960 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 21:02:22 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:22.882745 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.875617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.16.190.254:32967)
0927 21:02:22.875697 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.875698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.875754 (+ 56us) server_negotiation.cc:407] Connection header received
0927 21:02:22.875800 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.875800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.875809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.875861 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:22.876352 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.876950 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.877644 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.877851 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.877917 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.880153 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:22.880155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:22.880155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:22.880162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:22.881713 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.882031 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.882032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.882032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.882084 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:22.882290 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:22.882290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:22.882290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:22.882393 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:22.882469 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.882586 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.882674 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:02:22.884378 24656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:22.886094 24656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:22.892868 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.16.190.193:44073)
0927 21:02:22.888854 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.888859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.888962 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:02:22.889047 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.889049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.889071 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.889155 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:22.889463 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890648 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.891855 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892363 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892486 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892545 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.892701 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.892743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260927 21:02:22.892858 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:44073 (local address 127.0.0.1:60080)
0927 21:02:22.888598 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.888886 (+ 288us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.888961 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.889217 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.889221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.889223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:22.889303 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.889305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890750 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.890751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.891763 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.891765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892468 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.892560 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892574 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.892677 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.892743 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":20,"thread_start_us":47,"threads_started":1}
I20260927 21:02:22.892911 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.16.190.194:44659)
0927 21:02:22.888845 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.888850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.889104 (+ 254us) server_negotiation.cc:407] Connection header received
0927 21:02:22.889175 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.889178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.889202 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.889277 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:22.889488 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890224 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.891616 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892370 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892464 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892525 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.892731 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.892775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260927 21:02:22.893085 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46213 (local address 127.0.0.1:60650)
0927 21:02:22.888860 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.888890 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.888964 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.889422 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.889423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.889424 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:22.889480 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.889482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890739 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.890740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.891831 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.891834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892792 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.892888 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892892 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.892942 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.892990 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260927 21:02:22.893471 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.16.190.195:46213)
0927 21:02:22.889228 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.889231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.889241 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:22.889278 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.889279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.889293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.889362 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:22.889563 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890646 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.891903 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892611 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892708 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892759 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.893237 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.893356 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":129,"threads_started":1}
I20260927 21:02:22.893476 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.888448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.0.0.1:58970)
0927 21:02:22.889040 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.889047 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.889103 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.889351 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.889352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.889354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:22.889408 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.889410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.890314 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.890316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.891522 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.891524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.892560 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.892561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.892626 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.892631 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.892681 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.893414 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":647,"thread_start_us":83,"threads_started":1}
I20260927 21:02:22.893656 24794 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3932458204f04a9da2ec1c9064e77 (DEFAULT_TABLE table=test-table [id=cdea70bf77cf49ed83b131528c081bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:22.894088 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3932458204f04a9da2ec1c9064e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.894142 24935 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3932458204f04a9da2ec1c9064e77 (DEFAULT_TABLE table=test-table [id=cdea70bf77cf49ed83b131528c081bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:22.894529 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3932458204f04a9da2ec1c9064e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.897504 25169 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Bootstrap starting.
I20260927 21:02:22.897989 25075 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3932458204f04a9da2ec1c9064e77 (DEFAULT_TABLE table=test-table [id=cdea70bf77cf49ed83b131528c081bc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:22.898136 25169 tablet_bootstrap.cc:654] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:22.898315 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3932458204f04a9da2ec1c9064e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.898490 25169 log.cc:824] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:22.899180 25169 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: No bootstrap required, opened a new log
I20260927 21:02:22.899235 25169 ts_tablet_manager.cc:1397] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:22.901057 25170 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Bootstrap starting.
I20260927 21:02:22.900990 25169 raft_consensus.cc:348] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.901166 25169 raft_consensus.cc:374] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:22.901208 25169 raft_consensus.cc:729] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Initialized, Role: FOLLOWER
I20260927 21:02:22.901351 25169 consensus_queue.cc:260] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.901455 25172 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Bootstrap starting.
I20260927 21:02:22.901610 25169 ts_tablet_manager.cc:1428] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:02:22.901728 25170 tablet_bootstrap.cc:654] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:22.901739 24861 heartbeater.cc:503] Master 127.16.190.254:32967 was elected leader, sending a full tablet report...
I20260927 21:02:22.902046 25172 tablet_bootstrap.cc:654] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:22.902060 25170 log.cc:824] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:22.902376 25172 log.cc:824] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:22.902794 25170 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: No bootstrap required, opened a new log
I20260927 21:02:22.902866 25170 ts_tablet_manager.cc:1397] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:22.903190 25172 tablet_bootstrap.cc:492] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: No bootstrap required, opened a new log
I20260927 21:02:22.903240 25172 ts_tablet_manager.cc:1397] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:22.904699 25170 raft_consensus.cc:348] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.904897 25170 raft_consensus.cc:374] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:22.904943 25170 raft_consensus.cc:729] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Initialized, Role: FOLLOWER
I20260927 21:02:22.904937 25172 raft_consensus.cc:348] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.905079 25172 raft_consensus.cc:374] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:22.905119 25172 raft_consensus.cc:729] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Initialized, Role: FOLLOWER
I20260927 21:02:22.905100 25170 consensus_queue.cc:260] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.905241 25172 consensus_queue.cc:260] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.905414 25170 ts_tablet_manager.cc:1428] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:02:22.905427 25001 heartbeater.cc:503] Master 127.16.190.254:32967 was elected leader, sending a full tablet report...
I20260927 21:02:22.905486 25172 ts_tablet_manager.cc:1428] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:02:22.905596 25141 heartbeater.cc:503] Master 127.16.190.254:32967 was elected leader, sending a full tablet report...
W20260927 21:02:22.919400 25002 tablet.cc:2369] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:22.965090 25176 raft_consensus.cc:482] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:22.965220 25176 raft_consensus.cc:504] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.965559 25176 leader_election.cc:288] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:22.969601 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:44073 (local address 127.16.190.194:46965)
0927 21:02:22.965839 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.965851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.965896 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.966076 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.966077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.966079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:22.966156 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.966157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.967149 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.967150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.968425 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.968429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.969275 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.969276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.969388 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.969390 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.969456 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.969505 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":31}
I20260927 21:02:22.969630 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.965857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:46965 (local address 127.16.190.193:44073)
0927 21:02:22.965919 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.965921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.965935 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:02:22.965966 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.965967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.965978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.966048 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:22.966255 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.967008 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.968566 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.969171 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.969251 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.969260 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.969503 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.969556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:22.969630 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.965884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51943 (local address 127.16.190.195:46213)
0927 21:02:22.965929 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.965931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.966114 (+ 183us) server_negotiation.cc:407] Connection header received
0927 21:02:22.966147 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.966148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.966156 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.966194 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:22.966363 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.967047 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.968578 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.969161 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.969254 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.969262 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.969503 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.969556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:22.969615 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.965801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46213 (local address 127.16.190.194:51943)
0927 21:02:22.965990 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:22.966001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:22.966071 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:22.966224 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:22.966224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:22.966226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:22.966287 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.966288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.967149 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.967150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.968425 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:22.968429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.969275 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.969276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:22.969384 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.969389 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:22.969456 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:02:22.969541 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":91,"thread_start_us":107,"threads_started":1}
I20260927 21:02:22.970037 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a3932458204f04a9da2ec1c9064e77" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" is_pre_election: true
I20260927 21:02:22.970052 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a3932458204f04a9da2ec1c9064e77" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907" is_pre_election: true
I20260927 21:02:22.970216 24814 raft_consensus.cc:2454] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 0.
I20260927 21:02:22.970216 25095 raft_consensus.cc:2454] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 0.
I20260927 21:02:22.970474 24888 leader_election.cc:302] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909cf0089dba42999d75678728cf6cba, bcdb8f3af2e549fe8ab7b1afdcb05a6b; no voters:
I20260927 21:02:22.970721 25176 raft_consensus.cc:2781] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:22.970783 25176 raft_consensus.cc:482] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:22.970799 25176 raft_consensus.cc:3037] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:22.971740 25176 raft_consensus.cc:504] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.971949 25176 leader_election.cc:288] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 election: Requested vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:22.972095 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a3932458204f04a9da2ec1c9064e77" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907"
I20260927 21:02:22.972113 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a3932458204f04a9da2ec1c9064e77" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
I20260927 21:02:22.972182 25095 raft_consensus.cc:3037] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:22.972182 24814 raft_consensus.cc:3037] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:22.973124 25095 raft_consensus.cc:2454] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 1.
I20260927 21:02:22.973124 24814 raft_consensus.cc:2454] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 1.
I20260927 21:02:22.973299 24889 leader_election.cc:302] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:22.973441 25176 raft_consensus.cc:2781] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:22.973592 25176 raft_consensus.cc:686] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 1 LEADER]: Becoming Leader. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Running, Role: LEADER
I20260927 21:02:22.973702 25176 consensus_queue.cc:237] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:22.974689 24655 catalog_manager.cc:5495] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba reported cstate change: term changed from 0 to 1, leader changed from <none> to 909cf0089dba42999d75678728cf6cba (127.16.190.194). New cstate: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } health_report { overall_health: UNKNOWN } } }
I20260927 21:02:22.985831 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:22.983293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.16.190.194:44659)
0927 21:02:22.983356 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:22.983358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:22.983367 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:22.983405 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:22.983406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:22.983429 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:22.983511 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:22.983699 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.984191 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.985018 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:22.985229 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:22.985307 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:22.985488 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:22.985604 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:22.985699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:22.985719 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 21:02:22.985740 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:22.987972 25095 raft_consensus.cc:1261] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Refusing update from remote peer 909cf0089dba42999d75678728cf6cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:22.988008 24814 raft_consensus.cc:1261] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Refusing update from remote peer 909cf0089dba42999d75678728cf6cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:22.988298 25176 consensus_queue.cc:1035] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:22.988425 25180 consensus_queue.cc:1035] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:22.991261 25185 mvcc.cc:205] Tried to move back new op lower bound from 7334063894475923456 to 7334063894420963328. Current Snapshot: MvccSnapshot[applied={T|T < 7334063894475923456}]
I20260927 21:02:22.992951 24655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:22.993053 24655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:22.993836 24655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Sending DeleteTablet for 3 replicas of tablet c2a3932458204f04a9da2ec1c9064e77
I20260927 21:02:22.994171 24794 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3932458204f04a9da2ec1c9064e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:22 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60080
I20260927 21:02:22.994217 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3932458204f04a9da2ec1c9064e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:22 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60650
I20260927 21:02:22.994395 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3932458204f04a9da2ec1c9064e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:22 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260927 21:02:22.994612 25194 tablet_replica.cc:330] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: stopping tablet replica
I20260927 21:02:22.994724 25194 raft_consensus.cc:2229] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:22.994837 25194 raft_consensus.cc:2258] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:22.994822 24655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:22.994962 24655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:22.995327 25194 ts_tablet_manager.cc:1905] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:22.995378 25192 tablet_replica.cc:330] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: stopping tablet replica
I20260927 21:02:22.995467 25192 raft_consensus.cc:2229] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:22.995561 25192 raft_consensus.cc:2258] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:22.996346 24794 tablet_service.cc:1467] Processing CreateTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 (DEFAULT_TABLE table=test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:22.996476 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffce3dc16e4f48edbf8158c7df5f29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.997155 25194 ts_tablet_manager.cc:1918] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:22.997242 25194 log.cc:1196] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/c2a3932458204f04a9da2ec1c9064e77
I20260927 21:02:22.997491 25194 ts_tablet_manager.cc:1939] T c2a3932458204f04a9da2ec1c9064e77 P 909cf0089dba42999d75678728cf6cba: Deleting consensus metadata
I20260927 21:02:22.997953 24935 tablet_service.cc:1467] Processing CreateTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 (DEFAULT_TABLE table=test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:22.998059 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffce3dc16e4f48edbf8158c7df5f29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:22.999221 25170 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Bootstrap starting.
I20260927 21:02:22.999229 25169 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Bootstrap starting.
I20260927 21:02:22.999653 25170 tablet_bootstrap.cc:654] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.000001 25169 tablet_bootstrap.cc:654] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.000342 25170 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: No bootstrap required, opened a new log
I20260927 21:02:23.000396 25170 ts_tablet_manager.cc:1397] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.000474 25075 tablet_service.cc:1467] Processing CreateTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 (DEFAULT_TABLE table=test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.000556 25170 raft_consensus.cc:348] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.000638 25169 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: No bootstrap required, opened a new log
I20260927 21:02:23.000643 25170 raft_consensus.cc:374] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.000667 25170 raft_consensus.cc:729] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.000679 25169 ts_tablet_manager.cc:1397] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.000677 24641 catalog_manager.cc:4841] TS 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659): tablet c2a3932458204f04a9da2ec1c9064e77 (table test-table [id=cdea70bf77cf49ed83b131528c081bc8]) successfully deleted
I20260927 21:02:23.000715 25170 consensus_queue.cc:260] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.000837 25170 ts_tablet_manager.cc:1428] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:23.000849 25169 raft_consensus.cc:348] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.000921 25169 raft_consensus.cc:374] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.000941 25169 raft_consensus.cc:729] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.000980 25169 consensus_queue.cc:260] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.001070 25169 ts_tablet_manager.cc:1428] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:23.001101 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffce3dc16e4f48edbf8158c7df5f29a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.002673 25172 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Bootstrap starting.
I20260927 21:02:23.003115 25172 tablet_bootstrap.cc:654] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.003847 25192 ts_tablet_manager.cc:1905] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.004004 25172 tablet_bootstrap.cc:492] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: No bootstrap required, opened a new log
I20260927 21:02:23.004084 25172 ts_tablet_manager.cc:1397] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.004251 25172 raft_consensus.cc:348] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.004357 25172 raft_consensus.cc:374] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.004383 25172 raft_consensus.cc:729] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.004436 25172 consensus_queue.cc:260] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.004537 25172 ts_tablet_manager.cc:1428] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:23.004875 25193 tablet_replica.cc:330] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: stopping tablet replica
I20260927 21:02:23.005050 25193 raft_consensus.cc:2229] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.005237 25193 raft_consensus.cc:2258] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.005771 25193 ts_tablet_manager.cc:1905] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.005831 25192 ts_tablet_manager.cc:1918] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:23.005923 25192 log.cc:1196] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/c2a3932458204f04a9da2ec1c9064e77
I20260927 21:02:23.006235 25192 ts_tablet_manager.cc:1939] T c2a3932458204f04a9da2ec1c9064e77 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting consensus metadata
I20260927 21:02:23.006894 24643 catalog_manager.cc:4841] TS 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073): tablet c2a3932458204f04a9da2ec1c9064e77 (table test-table [id=cdea70bf77cf49ed83b131528c081bc8]) successfully deleted
I20260927 21:02:23.007651 25193 ts_tablet_manager.cc:1918] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:23.007750 25193 log.cc:1196] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/c2a3932458204f04a9da2ec1c9064e77
I20260927 21:02:23.008020 25193 ts_tablet_manager.cc:1939] T c2a3932458204f04a9da2ec1c9064e77 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting consensus metadata
I20260927 21:02:23.008476 24642 catalog_manager.cc:4841] TS bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213): tablet c2a3932458204f04a9da2ec1c9064e77 (table test-table [id=cdea70bf77cf49ed83b131528c081bc8]) successfully deleted
W20260927 21:02:23.009358 24862 tablet.cc:2369] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:02:23.091178 25142 tablet.cc:2369] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:23.127214 25177 raft_consensus.cc:482] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:23.127311 25177 raft_consensus.cc:504] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.127637 25177 leader_election.cc:288] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659)
I20260927 21:02:23.130607 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.127963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:48017 (local address 127.16.190.194:44659)
0927 21:02:23.128016 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.128018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.128030 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:23.128086 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.128087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.128099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.128167 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:23.128341 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.128886 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.129846 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.130226 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.130315 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.130324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.130482 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.130533 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:02:23.130621 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.127852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.16.190.195:48017)
0927 21:02:23.127918 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.127933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.128015 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.128184 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.128184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.128186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:23.128265 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.128267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.129007 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.129008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.129752 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.129754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.130310 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.130311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.130395 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.130398 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.130472 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.130517 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:02:23.130641 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.127975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:37099 (local address 127.16.190.193:44073)
0927 21:02:23.128053 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.128055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.128128 (+ 73us) server_negotiation.cc:407] Connection header received
0927 21:02:23.128177 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.128178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.128198 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.128241 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:23.128414 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.128951 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.129905 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.130276 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.130358 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.130366 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.130531 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.130567 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 21:02:23.130718 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.127852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:44073 (local address 127.16.190.195:37099)
0927 21:02:23.128042 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.128053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.128122 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.128254 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.128254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.128256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:23.128343 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.128344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.129040 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.129041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.129802 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.129803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.130369 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.130370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.130440 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.130443 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.130518 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.130556 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":58,"thread_start_us":82,"threads_started":1}
I20260927 21:02:23.130970 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffce3dc16e4f48edbf8158c7df5f29a0" candidate_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907" is_pre_election: true
I20260927 21:02:23.130934 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffce3dc16e4f48edbf8158c7df5f29a0" candidate_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba" is_pre_election: true
I20260927 21:02:23.131075 24814 raft_consensus.cc:2454] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcdb8f3af2e549fe8ab7b1afdcb05a6b in term 0.
I20260927 21:02:23.131090 24955 raft_consensus.cc:2454] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcdb8f3af2e549fe8ab7b1afdcb05a6b in term 0.
I20260927 21:02:23.131265 25029 leader_election.cc:302] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, bcdb8f3af2e549fe8ab7b1afdcb05a6b; no voters:
I20260927 21:02:23.131410 25177 raft_consensus.cc:2781] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:23.131474 25177 raft_consensus.cc:482] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:23.131517 25177 raft_consensus.cc:3037] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.132205 25177 raft_consensus.cc:504] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.132360 25177 leader_election.cc:288] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [CANDIDATE]: Term 1 election: Requested vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659)
I20260927 21:02:23.132524 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffce3dc16e4f48edbf8158c7df5f29a0" candidate_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907"
I20260927 21:02:23.132602 24814 raft_consensus.cc:3037] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.132535 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffce3dc16e4f48edbf8158c7df5f29a0" candidate_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba"
I20260927 21:02:23.132632 24955 raft_consensus.cc:3037] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.133246 24814 raft_consensus.cc:2454] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcdb8f3af2e549fe8ab7b1afdcb05a6b in term 1.
I20260927 21:02:23.133412 25029 leader_election.cc:302] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, bcdb8f3af2e549fe8ab7b1afdcb05a6b; no voters:
I20260927 21:02:23.133436 24955 raft_consensus.cc:2454] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcdb8f3af2e549fe8ab7b1afdcb05a6b in term 1.
I20260927 21:02:23.133530 25177 raft_consensus.cc:2781] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:23.133656 25177 raft_consensus.cc:686] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 LEADER]: Becoming Leader. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Running, Role: LEADER
I20260927 21:02:23.133740 25177 consensus_queue.cc:237] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.134473 24656 catalog_manager.cc:5495] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b reported cstate change: term changed from 0 to 1, leader changed from <none> to bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195). New cstate: current_term: 1 leader_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } health_report { overall_health: HEALTHY } } }
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542942, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.254@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:23.168799 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.161744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.16.190.254:32967)
0927 21:02:23.161843 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.161846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.161980 (+ 134us) server_negotiation.cc:407] Connection header received
0927 21:02:23.162067 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.162068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.162079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.162152 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:23.162648 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.163144 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.163896 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.164087 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.164173 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.166378 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:23.166379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:23.166379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:23.166385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:23.167596 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:23.167982 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:23.167983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:23.167983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:23.168036 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:23.168238 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:23.168239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:23.168239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:23.168350 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:23.168426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.168579 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.168687 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 21:02:23.168897 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.161672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:32967 (local address 127.0.0.1:59968)
0927 21:02:23.161899 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.161908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.162000 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.162157 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.162158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.162502 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:23.162556 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.162559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.163242 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.163244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.163807 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.163809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.164174 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.164175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.164226 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.164835 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:23.164839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:23.166258 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:23.167694 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:23.167702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:23.167704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:23.167896 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:23.168101 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:23.168102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:23.168102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:23.168170 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:23.168416 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:23.168424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:23.168469 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.168556 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.168748 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":84,"thread_start_us":94,"threads_started":1}
I20260927 21:02:23.170976 24656 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:59968:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 21:02:23.171201 24656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:23.172646 24794 tablet_service.cc:1467] Processing CreateTablet for tablet a3436f38607543e1b2e59d50fae79478 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f102d567b375438bb12430c756922578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:02:23.172670 24935 tablet_service.cc:1467] Processing CreateTablet for tablet a3436f38607543e1b2e59d50fae79478 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f102d567b375438bb12430c756922578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:02:23.172816 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3436f38607543e1b2e59d50fae79478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.172847 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3436f38607543e1b2e59d50fae79478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.174080 25170 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: Bootstrap starting.
I20260927 21:02:23.174175 25169 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: Bootstrap starting.
I20260927 21:02:23.174245 25075 tablet_service.cc:1467] Processing CreateTablet for tablet a3436f38607543e1b2e59d50fae79478 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f102d567b375438bb12430c756922578]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 21:02:23.174351 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3436f38607543e1b2e59d50fae79478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.174458 25170 tablet_bootstrap.cc:654] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.174525 25169 tablet_bootstrap.cc:654] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.175117 25170 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: No bootstrap required, opened a new log
I20260927 21:02:23.175171 25170 ts_tablet_manager.cc:1397] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.175182 25169 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: No bootstrap required, opened a new log
I20260927 21:02:23.175216 25169 ts_tablet_manager.cc:1397] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.175290 25170 raft_consensus.cc:348] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.175369 25170 raft_consensus.cc:374] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.175349 25169 raft_consensus.cc:348] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.175390 25170 raft_consensus.cc:729] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.175411 25169 raft_consensus.cc:374] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.175439 25169 raft_consensus.cc:729] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.175436 25170 consensus_queue.cc:260] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.175477 25169 consensus_queue.cc:260] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.175554 25169 ts_tablet_manager.cc:1428] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:23.175558 25170 ts_tablet_manager.cc:1428] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:23.175666 25176 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.175695 25173 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.175741 25176 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.175755 25172 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Bootstrap starting.
I20260927 21:02:23.175768 25173 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.176162 25172 tablet_bootstrap.cc:654] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.176945 25172 tablet_bootstrap.cc:492] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: No bootstrap required, opened a new log
I20260927 21:02:23.176985 25172 ts_tablet_manager.cc:1397] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.177296 25172 raft_consensus.cc:348] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.177358 25172 raft_consensus.cc:374] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.177371 25172 raft_consensus.cc:729] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.177425 25172 consensus_queue.cc:260] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.177546 25172 ts_tablet_manager.cc:1428] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:23.177595 25177 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.177671 25177 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.208967 25176 raft_consensus.cc:482] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:23.209097 25176 raft_consensus.cc:504] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.209286 25176 leader_election.cc:288] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.209470 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3436f38607543e1b2e59d50fae79478" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907" is_pre_election: true
I20260927 21:02:23.209496 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3436f38607543e1b2e59d50fae79478" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" is_pre_election: true
I20260927 21:02:23.209589 24814 raft_consensus.cc:2454] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 0.
I20260927 21:02:23.209638 25095 raft_consensus.cc:2454] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 0.
I20260927 21:02:23.209761 24889 leader_election.cc:302] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.209880 25176 raft_consensus.cc:2781] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:23.209949 25176 raft_consensus.cc:482] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:23.209980 25176 raft_consensus.cc:3037] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.210664 25176 raft_consensus.cc:504] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.210840 25176 leader_election.cc:288] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 election: Requested vote from peers 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.210976 24814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3436f38607543e1b2e59d50fae79478" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf66e60dc3a4ff1877b17a34086a907"
I20260927 21:02:23.211053 24814 raft_consensus.cc:3037] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.211042 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3436f38607543e1b2e59d50fae79478" candidate_uuid: "909cf0089dba42999d75678728cf6cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
I20260927 21:02:23.211119 25095 raft_consensus.cc:3037] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.211740 24814 raft_consensus.cc:2454] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 1.
I20260927 21:02:23.211915 24889 leader_election.cc:302] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.211943 25095 raft_consensus.cc:2454] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909cf0089dba42999d75678728cf6cba in term 1.
I20260927 21:02:23.212011 25176 raft_consensus.cc:2781] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:23.212087 25176 raft_consensus.cc:686] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [term 1 LEADER]: Becoming Leader. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Running, Role: LEADER
I20260927 21:02:23.212142 25176 consensus_queue.cc:237] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.212412 25180 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: TxnStatusTablet state changed. Reason: New leader 909cf0089dba42999d75678728cf6cba. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.212517 25180 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:02:23.212680 25211 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 21:02:23.212858 25211 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 21:02:23.212777 24656 catalog_manager.cc:5495] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba reported cstate change: term changed from 0 to 1, leader changed from <none> to 909cf0089dba42999d75678728cf6cba (127.16.190.194). New cstate: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } health_report { overall_health: UNKNOWN } } }
I20260927 21:02:23.224339 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.221427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.0.0.1:58982)
0927 21:02:23.221502 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.221515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.221566 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.221852 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.221853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.221854 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:02:23.221899 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.221900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.222792 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.222793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.223363 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.223365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.223857 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.223858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.223928 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.223945 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.224170 (+ 225us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.224181 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.224228 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.224275 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:02:23.224351 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.221651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.16.190.194:44659)
0927 21:02:23.221725 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.221727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.221736 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:23.221763 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.221764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.221775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.221834 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.221966 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.222710 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.223441 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.223753 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.223824 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.224037 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.224074 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.224128 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.224246 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.224269 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:02:23.225538 25095 raft_consensus.cc:1261] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Refusing update from remote peer 909cf0089dba42999d75678728cf6cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.225584 24814 raft_consensus.cc:1261] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Refusing update from remote peer 909cf0089dba42999d75678728cf6cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.225806 25180 consensus_queue.cc:1035] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.225911 25176 consensus_queue.cc:1035] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.226420 25177 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: TxnStatusTablet state changed. Reason: New leader 909cf0089dba42999d75678728cf6cba. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.226511 25177 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.226974 25180 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.227057 25180 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:02:23.227115 25183 mvcc.cc:205] Tried to move back new op lower bound from 7334063895449903104 to 7334063895397429248. Current Snapshot: MvccSnapshot[applied={T|T < 7334063895449903104 or (T in {7334063895449903104})}]
I20260927 21:02:23.227626 25173 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: TxnStatusTablet state changed. Reason: New leader 909cf0089dba42999d75678728cf6cba. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.227725 25173 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.227805 25177 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.227871 25177 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.228677 25176 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.228757 25176 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 909cf0089dba42999d75678728cf6cba: This TxnStatusTablet replica's current role is: LEADER
I20260927 21:02:23.229228 25184 mvcc.cc:205] Tried to move back new op lower bound from 7334063895449903104 to 7334063895397429248. Current Snapshot: MvccSnapshot[applied={T|T < 7334063895449903104 or (T in {7334063895449903104})}]
I20260927 21:02:23.229439 25173 tablet_replica.cc:439] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "909cf0089dba42999d75678728cf6cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } } }
I20260927 21:02:23.229506 25173 tablet_replica.cc:442] T a3436f38607543e1b2e59d50fae79478 P 5bf66e60dc3a4ff1877b17a34086a907: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 21:02:23.232420 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.229620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.16.190.195:46213)
0927 21:02:23.229663 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.229665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.229672 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:23.229709 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.229710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.229731 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.229799 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.229988 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.230740 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.231491 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.231814 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.231889 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.232008 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.232151 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.232227 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.232293 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.232315 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:02:23.238220 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.235200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:60781 (local address 127.16.190.194:44659)
0927 21:02:23.235251 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.235253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.235357 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:02:23.235420 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.235421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.235436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.235499 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.235642 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.236374 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.237522 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.237731 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.237792 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.237998 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.238025 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.238058 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.238126 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.238158 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:23.238251 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.235150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.16.190.195:60781)
0927 21:02:23.235269 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.235296 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.235367 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.235502 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.235502 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.235503 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:02:23.235571 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.235572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.236473 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.236474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.237410 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.237413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.237814 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.237815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.237888 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.237911 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.238073 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.238075 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.238123 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.238169 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 21:02:23.240737 24955 raft_consensus.cc:1261] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Refusing update from remote peer bcdb8f3af2e549fe8ab7b1afdcb05a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.240988 24814 raft_consensus.cc:1261] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Refusing update from remote peer bcdb8f3af2e549fe8ab7b1afdcb05a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.241158 25177 consensus_queue.cc:1035] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.241297 25199 consensus_queue.cc:1035] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.250392 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.247464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:54171 (local address 127.16.190.195:46213)
0927 21:02:23.247506 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.247508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.247589 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:02:23.247612 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.247613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.247623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.247666 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.247841 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.248497 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.249659 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.249848 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.249912 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.250146 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.250185 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.250241 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.250310 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.250331 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:02:23.250656 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.247257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46213 (local address 127.16.190.194:54171)
0927 21:02:23.247485 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.247498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.247563 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.247678 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.247679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.247681 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:02:23.247769 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.247771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.248588 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.248590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.249561 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.249564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.249930 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.249931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.250009 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.250029 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.250259 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.250261 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.250308 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.250359 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260927 21:02:23.259889 24655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.260020 24655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.260737 24655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Sending DeleteTablet for 3 replicas of tablet ffce3dc16e4f48edbf8158c7df5f29a0
I20260927 21:02:23.261013 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260927 21:02:23.261102 24794 tablet_service.cc:1514] Processing DeleteTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60080
I20260927 21:02:23.261138 25194 tablet_replica.cc:330] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: stopping tablet replica
I20260927 21:02:23.261206 25194 raft_consensus.cc:2229] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.261211 25192 tablet_replica.cc:330] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: stopping tablet replica
I20260927 21:02:23.261257 25194 raft_consensus.cc:2258] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.261276 25192 raft_consensus.cc:2229] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.261276 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet ffce3dc16e4f48edbf8158c7df5f29a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60650
I20260927 21:02:23.261317 25192 raft_consensus.cc:2258] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.261361 25193 tablet_replica.cc:330] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: stopping tablet replica
I20260927 21:02:23.261421 25193 raft_consensus.cc:2229] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:23.261531 25193 raft_consensus.cc:2258] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.261688 25194 ts_tablet_manager.cc:1905] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.263250 25193 ts_tablet_manager.cc:1905] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.263908 25192 ts_tablet_manager.cc:1905] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.264791 25193 ts_tablet_manager.cc:1918] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:02:23.264907 25193 log.cc:1196] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/ffce3dc16e4f48edbf8158c7df5f29a0
I20260927 21:02:23.265189 25193 ts_tablet_manager.cc:1939] T ffce3dc16e4f48edbf8158c7df5f29a0 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting consensus metadata
I20260927 21:02:23.265494 25192 ts_tablet_manager.cc:1918] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:02:23.265560 25192 log.cc:1196] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/ffce3dc16e4f48edbf8158c7df5f29a0
I20260927 21:02:23.265627 25194 ts_tablet_manager.cc:1918] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:02:23.265692 25194 log.cc:1196] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/ffce3dc16e4f48edbf8158c7df5f29a0
I20260927 21:02:23.265784 25192 ts_tablet_manager.cc:1939] T ffce3dc16e4f48edbf8158c7df5f29a0 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting consensus metadata
I20260927 21:02:23.265908 25194 ts_tablet_manager.cc:1939] T ffce3dc16e4f48edbf8158c7df5f29a0 P 909cf0089dba42999d75678728cf6cba: Deleting consensus metadata
I20260927 21:02:23.265977 24642 catalog_manager.cc:4841] TS bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213): tablet ffce3dc16e4f48edbf8158c7df5f29a0 (table test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]) successfully deleted
I20260927 21:02:23.266320 24643 catalog_manager.cc:4841] TS 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073): tablet ffce3dc16e4f48edbf8158c7df5f29a0 (table test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]) successfully deleted
I20260927 21:02:23.266402 24641 catalog_manager.cc:4841] TS 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659): tablet ffce3dc16e4f48edbf8158c7df5f29a0 (table test-table [id=ae7a55d6b5594e0ab9f03db4e35059f9]) successfully deleted
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[24611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:02:23.299695 24706 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:23.292203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.16.190.254:32967)
0927 21:02:23.292251 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.292253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.292436 (+ 183us) server_negotiation.cc:407] Connection header received
0927 21:02:23.292498 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.292499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.292513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.292586 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:23.293330 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.294268 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.295518 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.295843 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.295912 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.299612 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59984: BlockingRecv error: recv got EOF from 127.0.0.1:59984 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[24611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM
I20260927 21:02:23.309847 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.300449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.16.190.254:32967)
0927 21:02:23.300753 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.300755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.301069 (+ 314us) server_negotiation.cc:407] Connection header received
0927 21:02:23.301121 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.301122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.301133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.301197 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:23.301803 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.302595 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.303585 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.303908 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.303983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.306810 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:23.306812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:23.306812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:23.306820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:23.308547 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:23.308982 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:23.308983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:23.308984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:23.309051 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:23.309282 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:23.309283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:23.309283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:23.309431 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:23.309539 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.309660 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.309752 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260927 21:02:23.311808 24655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:23.311998 24655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:23.313534 24935 tablet_service.cc:1467] Processing CreateTablet for tablet c66440ea69f44b4284309ca531588e60 (DEFAULT_TABLE table=test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.313565 24794 tablet_service.cc:1467] Processing CreateTablet for tablet c66440ea69f44b4284309ca531588e60 (DEFAULT_TABLE table=test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.313689 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66440ea69f44b4284309ca531588e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.313719 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66440ea69f44b4284309ca531588e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.315165 25169 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Bootstrap starting.
I20260927 21:02:23.315166 25170 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Bootstrap starting.
I20260927 21:02:23.315496 25170 tablet_bootstrap.cc:654] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.315570 25169 tablet_bootstrap.cc:654] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.315987 25075 tablet_service.cc:1467] Processing CreateTablet for tablet c66440ea69f44b4284309ca531588e60 (DEFAULT_TABLE table=test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.316103 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66440ea69f44b4284309ca531588e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.316684 25170 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: No bootstrap required, opened a new log
I20260927 21:02:23.316751 25170 ts_tablet_manager.cc:1397] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:23.317078 25170 raft_consensus.cc:348] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.317166 25170 raft_consensus.cc:374] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.317188 25170 raft_consensus.cc:729] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.317229 25170 consensus_queue.cc:260] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.317318 25170 ts_tablet_manager.cc:1428] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:23.317396 25172 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Bootstrap starting.
I20260927 21:02:23.317822 25172 tablet_bootstrap.cc:654] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.318919 25172 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: No bootstrap required, opened a new log
I20260927 21:02:23.318975 25172 ts_tablet_manager.cc:1397] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:02:23.319109 25172 raft_consensus.cc:348] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.319182 25172 raft_consensus.cc:374] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.319203 25172 raft_consensus.cc:729] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.319245 25172 consensus_queue.cc:260] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.319340 25172 ts_tablet_manager.cc:1428] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:23.319895 25169 tablet_bootstrap.cc:492] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: No bootstrap required, opened a new log
I20260927 21:02:23.319947 25169 ts_tablet_manager.cc:1397] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:02:23.320088 25169 raft_consensus.cc:348] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.320156 25169 raft_consensus.cc:374] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.320178 25169 raft_consensus.cc:729] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.320214 25169 consensus_queue.cc:260] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.320294 25169 ts_tablet_manager.cc:1428] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:23.355105 25173 raft_consensus.cc:482] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:23.355234 25173 raft_consensus.cc:504] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.355618 25173 leader_election.cc:288] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.359939 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.356018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:41769 (local address 127.16.190.195:46213)
0927 21:02:23.356063 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.356065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.356226 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:02:23.356336 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.356337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.356346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.356376 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:23.356613 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.357355 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.358970 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.359500 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.359554 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.359578 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.359823 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.359861 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:02:23.360451 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.355838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46213 (local address 127.16.190.193:41769)
0927 21:02:23.356155 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.356166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.356255 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.356440 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.356441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.356443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:23.356516 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.356519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.357465 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.357466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.358856 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.358860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.359658 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.359659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.359731 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.359735 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.360293 (+ 558us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.360345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":43,"thread_start_us":84,"threads_started":1}
I20260927 21:02:23.360884 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.355991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:59911 (local address 127.16.190.194:44659)
0927 21:02:23.356036 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.356038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.356196 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:02:23.356327 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.356328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.356337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.356368 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:23.356631 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.357369 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.359537 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.360054 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.360098 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.360106 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.360747 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.360791 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:02:23.360888 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66440ea69f44b4284309ca531588e60" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" is_pre_election: true
I20260927 21:02:23.360985 25095 raft_consensus.cc:2454] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 0.
I20260927 21:02:23.361081 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.355813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.16.190.193:59911)
0927 21:02:23.356127 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.356138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.356243 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.356438 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.356438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.356440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:23.356553 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.356555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.357498 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.357499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.359428 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.359431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.360163 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.360164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.360662 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.360667 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.360964 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.361005 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":66,"thread_start_us":75,"threads_started":1}
I20260927 21:02:23.361304 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66440ea69f44b4284309ca531588e60" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba" is_pre_election: true
I20260927 21:02:23.361379 24955 raft_consensus.cc:2454] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 0.
I20260927 21:02:23.361526 24746 leader_election.cc:302] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.361649 25173 raft_consensus.cc:2781] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:23.361678 25173 raft_consensus.cc:482] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:23.361697 25173 raft_consensus.cc:3037] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.362587 25173 raft_consensus.cc:504] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.362730 25173 leader_election.cc:288] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 election: Requested vote from peers 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.362902 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66440ea69f44b4284309ca531588e60" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
I20260927 21:02:23.362908 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66440ea69f44b4284309ca531588e60" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba"
I20260927 21:02:23.362977 24955 raft_consensus.cc:3037] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.362977 25095 raft_consensus.cc:3037] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.363771 24955 raft_consensus.cc:2454] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 1.
I20260927 21:02:23.363862 25095 raft_consensus.cc:2454] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 1.
I20260927 21:02:23.363931 24746 leader_election.cc:302] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.364001 25173 raft_consensus.cc:2781] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:23.364131 25173 raft_consensus.cc:686] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Running, Role: LEADER
I20260927 21:02:23.364185 25173 consensus_queue.cc:237] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.364908 24655 catalog_manager.cc:5495] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "5bf66e60dc3a4ff1877b17a34086a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } health_report { overall_health: UNKNOWN } } }
I20260927 21:02:23.376180 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.372475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.16.190.193:44073)
0927 21:02:23.372523 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.372525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.372532 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:23.372583 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.372584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.372610 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.372672 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.372930 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.373749 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.374939 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.375330 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.375395 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.375625 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.375764 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.375830 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.376066 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.376089 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:02:23.377415 25095 raft_consensus.cc:1261] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Refusing update from remote peer 5bf66e60dc3a4ff1877b17a34086a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.377758 24955 raft_consensus.cc:1261] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Refusing update from remote peer 5bf66e60dc3a4ff1877b17a34086a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.377807 25244 consensus_queue.cc:1035] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.378064 25173 consensus_queue.cc:1035] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.381531 24656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.381641 24656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.382254 24656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Sending DeleteTablet for 3 replicas of tablet c66440ea69f44b4284309ca531588e60
I20260927 21:02:23.382511 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet c66440ea69f44b4284309ca531588e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60650
I20260927 21:02:23.382593 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet c66440ea69f44b4284309ca531588e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260927 21:02:23.382630 25193 tablet_replica.cc:330] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: stopping tablet replica
I20260927 21:02:23.382679 25193 raft_consensus.cc:2229] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.382691 24794 tablet_service.cc:1514] Processing DeleteTablet for tablet c66440ea69f44b4284309ca531588e60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60080
I20260927 21:02:23.382726 25193 raft_consensus.cc:2258] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.382777 25194 tablet_replica.cc:330] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: stopping tablet replica
I20260927 21:02:23.382835 25194 raft_consensus.cc:2229] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.382884 25194 raft_consensus.cc:2258] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.382951 25192 tablet_replica.cc:330] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: stopping tablet replica
I20260927 21:02:23.382998 25192 raft_consensus.cc:2229] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:23.383085 25192 raft_consensus.cc:2258] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.383435 25194 ts_tablet_manager.cc:1905] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.383528 25193 ts_tablet_manager.cc:1905] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.385174 25194 ts_tablet_manager.cc:1918] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:23.385202 25193 ts_tablet_manager.cc:1918] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:23.385246 25193 log.cc:1196] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/c66440ea69f44b4284309ca531588e60
I20260927 21:02:23.385246 25194 log.cc:1196] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/c66440ea69f44b4284309ca531588e60
I20260927 21:02:23.385463 25194 ts_tablet_manager.cc:1939] T c66440ea69f44b4284309ca531588e60 P 909cf0089dba42999d75678728cf6cba: Deleting consensus metadata
I20260927 21:02:23.385463 25193 ts_tablet_manager.cc:1939] T c66440ea69f44b4284309ca531588e60 P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting consensus metadata
I20260927 21:02:23.385608 24656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:23.385771 24656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:23.385887 24642 catalog_manager.cc:4841] TS bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213): tablet c66440ea69f44b4284309ca531588e60 (table test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]) successfully deleted
I20260927 21:02:23.386088 25192 ts_tablet_manager.cc:1905] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.387259 24641 catalog_manager.cc:4841] TS 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659): tablet c66440ea69f44b4284309ca531588e60 (table test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]) successfully deleted
I20260927 21:02:23.388129 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 449bae1d0e394bccb589740656cbf33e (DEFAULT_TABLE table=test-table [id=f604a71d2b0e4f218650a978c9161a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.388128 24794 tablet_service.cc:1467] Processing CreateTablet for tablet 449bae1d0e394bccb589740656cbf33e (DEFAULT_TABLE table=test-table [id=f604a71d2b0e4f218650a978c9161a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.388244 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449bae1d0e394bccb589740656cbf33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.388366 24935 tablet_service.cc:1467] Processing CreateTablet for tablet 449bae1d0e394bccb589740656cbf33e (DEFAULT_TABLE table=test-table [id=f604a71d2b0e4f218650a978c9161a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:23.388459 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449bae1d0e394bccb589740656cbf33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.389591 25172 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Bootstrap starting.
I20260927 21:02:23.389725 25170 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Bootstrap starting.
I20260927 21:02:23.390014 25170 tablet_bootstrap.cc:654] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.390249 25172 tablet_bootstrap.cc:654] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.390522 25192 ts_tablet_manager.cc:1918] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:23.390609 25192 log.cc:1196] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/c66440ea69f44b4284309ca531588e60
I20260927 21:02:23.390822 25192 ts_tablet_manager.cc:1939] T c66440ea69f44b4284309ca531588e60 P 5bf66e60dc3a4ff1877b17a34086a907: Deleting consensus metadata
I20260927 21:02:23.390522 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449bae1d0e394bccb589740656cbf33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:23.391165 24643 catalog_manager.cc:4841] TS 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073): tablet c66440ea69f44b4284309ca531588e60 (table test-table [id=ec1ee5f73e6949e3b36c8d605bc3924a]) successfully deleted
I20260927 21:02:23.392014 25169 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: Bootstrap starting.
I20260927 21:02:23.392455 25169 tablet_bootstrap.cc:654] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:23.392674 25170 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: No bootstrap required, opened a new log
I20260927 21:02:23.392730 25170 ts_tablet_manager.cc:1397] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260927 21:02:23.392872 25170 raft_consensus.cc:348] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.392961 25170 raft_consensus.cc:374] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.392987 25170 raft_consensus.cc:729] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909cf0089dba42999d75678728cf6cba, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.393061 25170 consensus_queue.cc:260] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.393193 25170 ts_tablet_manager.cc:1428] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:23.393455 25169 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: No bootstrap required, opened a new log
I20260927 21:02:23.393498 25169 ts_tablet_manager.cc:1397] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:23.393652 25169 raft_consensus.cc:348] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.393721 25169 raft_consensus.cc:374] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.393743 25169 raft_consensus.cc:729] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.393780 25169 consensus_queue.cc:260] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.393883 25169 ts_tablet_manager.cc:1428] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:23.394631 25172 tablet_bootstrap.cc:492] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: No bootstrap required, opened a new log
I20260927 21:02:23.394698 25172 ts_tablet_manager.cc:1397] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 21:02:23.394860 25172 raft_consensus.cc:348] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.394955 25172 raft_consensus.cc:374] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:23.394984 25172 raft_consensus.cc:729] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcdb8f3af2e549fe8ab7b1afdcb05a6b, State: Initialized, Role: FOLLOWER
I20260927 21:02:23.395025 25172 consensus_queue.cc:260] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.395116 25172 ts_tablet_manager.cc:1428] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:23.419332 25244 raft_consensus.cc:482] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:23.419462 25244 raft_consensus.cc:504] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.419759 25244 leader_election.cc:288] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.419979 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449bae1d0e394bccb589740656cbf33e" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba" is_pre_election: true
I20260927 21:02:23.420080 24955 raft_consensus.cc:2454] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 0.
I20260927 21:02:23.420051 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449bae1d0e394bccb589740656cbf33e" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" is_pre_election: true
I20260927 21:02:23.420135 25095 raft_consensus.cc:2454] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 0.
I20260927 21:02:23.420253 24746 leader_election.cc:302] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.420353 25244 raft_consensus.cc:2781] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:23.420389 25244 raft_consensus.cc:482] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:23.420419 25244 raft_consensus.cc:3037] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.421239 25244 raft_consensus.cc:504] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.421393 25244 leader_election.cc:288] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 election: Requested vote from peers 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659), bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213)
I20260927 21:02:23.421586 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449bae1d0e394bccb589740656cbf33e" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909cf0089dba42999d75678728cf6cba"
I20260927 21:02:23.421614 25095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449bae1d0e394bccb589740656cbf33e" candidate_uuid: "5bf66e60dc3a4ff1877b17a34086a907" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b"
I20260927 21:02:23.421674 24955 raft_consensus.cc:3037] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.421682 25095 raft_consensus.cc:3037] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:23.422434 24955 raft_consensus.cc:2454] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 1.
I20260927 21:02:23.422434 25095 raft_consensus.cc:2454] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bf66e60dc3a4ff1877b17a34086a907 in term 1.
I20260927 21:02:23.422611 24746 leader_election.cc:302] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bf66e60dc3a4ff1877b17a34086a907, 909cf0089dba42999d75678728cf6cba; no voters:
I20260927 21:02:23.422756 25244 raft_consensus.cc:2781] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:23.422847 25244 raft_consensus.cc:686] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf66e60dc3a4ff1877b17a34086a907, State: Running, Role: LEADER
I20260927 21:02:23.422900 25244 consensus_queue.cc:237] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } }
I20260927 21:02:23.423548 24656 catalog_manager.cc:5495] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "5bf66e60dc3a4ff1877b17a34086a907" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bf66e60dc3a4ff1877b17a34086a907" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 44073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 } health_report { overall_health: UNKNOWN } } }
I20260927 21:02:23.437496 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.434422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:37981 (local address 127.16.190.194:44659)
0927 21:02:23.434474 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.434476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.434583 (+ 107us) server_negotiation.cc:407] Connection header received
0927 21:02:23.434657 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.434658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.434667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.434733 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.434908 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.435392 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.436536 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.436752 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.436872 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.437113 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.437144 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.437263 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.437315 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.437373 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:02:23.437531 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.434329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44659 (local address 127.16.190.193:37981)
0927 21:02:23.434518 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.434531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.434599 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.434755 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.434756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.434758 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:02:23.434836 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.434838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.435473 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.435474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.436446 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.436449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.436889 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.436890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.436993 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.437014 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.437247 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.437250 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.437308 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.437373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260927 21:02:23.440200 24955 raft_consensus.cc:1261] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Refusing update from remote peer 5bf66e60dc3a4ff1877b17a34086a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.440383 25095 raft_consensus.cc:1261] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Refusing update from remote peer 5bf66e60dc3a4ff1877b17a34086a907: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:23.440472 25173 consensus_queue.cc:1035] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909cf0089dba42999d75678728cf6cba" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.440599 25244 consensus_queue.cc:1035] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcdb8f3af2e549fe8ab7b1afdcb05a6b" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:23.448482 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.445567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:56507 (local address 127.16.190.193:44073)
0927 21:02:23.445611 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:23.445613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:23.445639 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:02:23.445737 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:23.445738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:23.445749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:23.445801 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:23.445967 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.446716 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.447658 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.447923 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.447989 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.448172 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.448202 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.448291 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:23.448357 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:02:23.448395 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:02:23.448496 25222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.445534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:44073 (local address 127.16.190.194:56507)
0927 21:02:23.445592 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.445605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:23.445655 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:23.445847 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:23.445847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:23.445849 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 21:02:23.445896 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.445898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.446796 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.446797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.447543 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:23.447546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:23.448025 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:23.448026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:23.448083 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:23.448091 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 21:02:23.448296 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 21:02:23.448298 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:02:23.448342 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:02:23.448396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 21:02:23.454159 24655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.454272 24655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:23.455041 24655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c50513fd02c9480ca9b366f2714b9428: Sending DeleteTablet for 3 replicas of tablet 449bae1d0e394bccb589740656cbf33e
I20260927 21:02:23.455348 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet 449bae1d0e394bccb589740656cbf33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60650
I20260927 21:02:23.455371 24794 tablet_service.cc:1514] Processing DeleteTablet for tablet 449bae1d0e394bccb589740656cbf33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:60080
I20260927 21:02:23.455454 25192 tablet_replica.cc:330] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: stopping tablet replica
I20260927 21:02:23.455462 25193 tablet_replica.cc:330] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: stopping tablet replica
I20260927 21:02:23.455461 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet 449bae1d0e394bccb589740656cbf33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:23 UTC) from {username='oryx', principal='oryx/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58970
I20260927 21:02:23.455500 25192 raft_consensus.cc:2229] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:23.455524 25193 raft_consensus.cc:2229] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.455538 25194 tablet_replica.cc:330] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: stopping tablet replica
I20260927 21:02:23.455559 25192 raft_consensus.cc:2258] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.455561 25193 raft_consensus.cc:2258] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.455585 25194 raft_consensus.cc:2229] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:23.455632 25194 raft_consensus.cc:2258] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:23.455842 25193 ts_tablet_manager.cc:1905] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.456064 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 24729
I20260927 21:02:23.456099 25194 ts_tablet_manager.cc:1905] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.456153 25192 ts_tablet_manager.cc:1905] T 449bae1d0e394bccb589740656cbf33e P 5bf66e60dc3a4ff1877b17a34086a907: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:23.457616 25194 ts_tablet_manager.cc:1918] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:02:23.457670 25194 log.cc:1196] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/449bae1d0e394bccb589740656cbf33e
I20260927 21:02:23.457739 25193 ts_tablet_manager.cc:1918] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 21:02:23.457795 25193 log.cc:1196] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/449bae1d0e394bccb589740656cbf33e
I20260927 21:02:23.457880 25194 ts_tablet_manager.cc:1939] T 449bae1d0e394bccb589740656cbf33e P 909cf0089dba42999d75678728cf6cba: Deleting consensus metadata
I20260927 21:02:23.458039 25193 ts_tablet_manager.cc:1939] T 449bae1d0e394bccb589740656cbf33e P bcdb8f3af2e549fe8ab7b1afdcb05a6b: Deleting consensus metadata
I20260927 21:02:23.458240 24641 catalog_manager.cc:4841] TS 909cf0089dba42999d75678728cf6cba (127.16.190.194:44659): tablet 449bae1d0e394bccb589740656cbf33e (table test-table [id=f604a71d2b0e4f218650a978c9161a76]) successfully deleted
I20260927 21:02:23.458390 24642 catalog_manager.cc:4841] TS bcdb8f3af2e549fe8ab7b1afdcb05a6b (127.16.190.195:46213): tablet 449bae1d0e394bccb589740656cbf33e (table test-table [id=f604a71d2b0e4f218650a978c9161a76]) successfully deleted
W20260927 21:02:23.462620 24643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:44073 (error 108)
I20260927 21:02:23.462884 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 24870
I20260927 21:02:23.468895 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:23.463026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:44073 (local address 127.0.0.1:60096)
0927 21:02:23.468732 (+ 5706us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:23.468786 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5638,"thread_start_us":40,"threads_started":1}
I20260927 21:02:23.468890 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25010
W20260927 21:02:23.470176 24643 catalog_manager.cc:4584] TS 5bf66e60dc3a4ff1877b17a34086a907 (127.16.190.193:44073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 449bae1d0e394bccb589740656cbf33e: Network error: Client connection negotiation failed: client connection to 127.16.190.193:44073: connect: Connection refused (error 111)
I20260927 21:02:23.474720 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 24624
2026-09-27T21:02:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2448 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[25264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[25264](info): set up 2 sockets
Sep 27 21:02:23 dist-test-slave-6c7s krb5kdc[25264](info): commencing operation
krb5kdc: starting...
W20260927 21:02:24.530237 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:24 dist-test-slave-6c7s krb5kdc[25264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:24.652927 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42961
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:24.734615 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:24.734896 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:24.734933 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:24.736076 25277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:24.736157 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:24.736182 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:24.736204 25277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:24.736232 25277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:24.737776 25277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:24.738067 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:24.738304 25277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:24.743907 25283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:24.743914 25285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:24.743933 25277 server_base.cc:1034] running on GCE node
W20260927 21:02:24.743914 25282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:24.744393 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:24.744874 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:24.746075 25277 hybrid_clock.cc:648] HybridClock initialized: now 1790542944746041 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T21:02:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1274 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:24 dist-test-slave-6c7s krb5kdc[25290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:24 dist-test-slave-6c7s krb5kdc[25290](info): set up 2 sockets
Sep 27 21:02:24 dist-test-slave-6c7s krb5kdc[25290](info): commencing operation
krb5kdc: starting...
W20260927 21:02:25.798398 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:25 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:25.924325 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39099
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.005648 25303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.005916 25303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.005954 25303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.007130 25303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:26.007212 25303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.007237 25303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.007267 25303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:26.007298 25303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:26.008801 25303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.009099 25303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.009341 25303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.014238 25308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.014232 25309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.014232 25311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.014582 25303 server_base.cc:1034] running on GCE node
I20260927 21:02:26.014882 25303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.015167 25303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.016332 25303 hybrid_clock.cc:648] HybridClock initialized: now 1790542946016311 us; error 42 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.023270 25303 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:26.023869 25303 webserver.cc:460] Webserver started at http://127.16.190.254:33705/ using document root <none> and password file <none>
I20260927 21:02:26.024184 25303 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.024255 25303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.024397 25303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:26.025772 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "9d59d94683e34ecab75796a0ad5105d6"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "e8de0d1cd15eea285c84897c5a8b2be4"
server_key_iv: "b4c3ae416977af36b287e878704747fe"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.026193 25303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "9d59d94683e34ecab75796a0ad5105d6"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "e8de0d1cd15eea285c84897c5a8b2be4"
server_key_iv: "b4c3ae416977af36b287e878704747fe"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.028072 25303 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 21:02:26.028919 25318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.029083 25303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.029176 25303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "9d59d94683e34ecab75796a0ad5105d6"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "e8de0d1cd15eea285c84897c5a8b2be4"
server_key_iv: "b4c3ae416977af36b287e878704747fe"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.029240 25303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.050130 25303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.052217 25303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.052336 25303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.056116 25303 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39099
I20260927 21:02:26.056174 25370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39099 every 8 connection(s)
I20260927 21:02:26.056516 25303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:26.057448 25371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:26.059636 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25303
I20260927 21:02:26.059736 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:26.059846 25371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Bootstrap starting.
I20260927 21:02:26.059958 17147 external_mini_cluster.cc:1428] Setting key c2f42736fb74c00276aea35670a101ce
I20260927 21:02:26.060604 25371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:26.060956 25371 log.cc:824] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:26.061679 25371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: No bootstrap required, opened a new log
I20260927 21:02:26.063501 25371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.063674 25371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:26.063721 25371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d59d94683e34ecab75796a0ad5105d6, State: Initialized, Role: FOLLOWER
I20260927 21:02:26.063858 25371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.063958 25371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:26.064009 25371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:26.064057 25371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:26.064662 25371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.064790 25371 leader_election.cc:302] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d59d94683e34ecab75796a0ad5105d6; no voters:
I20260927 21:02:26.065030 25371 leader_election.cc:288] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:26.065049 25376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:26.065171 25376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 LEADER]: Becoming Leader. State: Replica: 9d59d94683e34ecab75796a0ad5105d6, State: Running, Role: LEADER
I20260927 21:02:26.065263 25376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.065320 25371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:26.065619 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d59d94683e34ecab75796a0ad5105d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } } }
I20260927 21:02:26.065696 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:26.065829 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d59d94683e34ecab75796a0ad5105d6. Latest consensus state: current_term: 1 leader_uuid: "9d59d94683e34ecab75796a0ad5105d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } } }
I20260927 21:02:26.065894 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:26.066618 25382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:26.067261 25382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:26.069180 25382 catalog_manager.cc:1347] Generated new cluster ID: 33bb7cd9d2054fde945a008842327ba4
I20260927 21:02:26.069271 25382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:26.073057 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.060586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.16.190.254:39099)
0927 21:02:26.060801 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.060857 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.061299 (+ 442us) server_negotiation.cc:407] Connection header received
0927 21:02:26.061930 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.061939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.061995 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.062162 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.062871 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.063940 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.064919 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.065621 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.065738 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.069152 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:26.069156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:26.069157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:26.069170 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:26.071257 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.071683 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.071687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.071687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.071805 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.072041 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.072042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.072042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.072311 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:26.072509 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:26.072618 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:02:26.072779 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260927 21:02:26.084662 25382 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:26.085304 25382 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:26.095901 25382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Generated new TSK 0
I20260927 21:02:26.096134 25382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:26.124274 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.207262 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.207499 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.207526 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.208864 25397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.208909 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.208935 25397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:26.210448 25397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.210726 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.210979 25397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.216190 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.216199 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.216235 25397 server_base.cc:1034] running on GCE node
W20260927 21:02:26.216203 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.216605 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.216922 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.218098 25397 hybrid_clock.cc:648] HybridClock initialized: now 1790542946218075 us; error 41 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.221041 25397 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:26.221585 25397 webserver.cc:460] Webserver started at http://127.16.190.193:38335/ using document root <none> and password file <none>
I20260927 21:02:26.221829 25397 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.221901 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.222060 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:26.223186 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "949c2e7084c948a5baf753a42377e6ea"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "9fb8edd7f209d8c82ad9201ba5c3fe09"
server_key_iv: "2673d5b684954cfccf99e9d5c1c574f6"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.223575 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "949c2e7084c948a5baf753a42377e6ea"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "9fb8edd7f209d8c82ad9201ba5c3fe09"
server_key_iv: "2673d5b684954cfccf99e9d5c1c574f6"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.225126 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:02:26.225965 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.226173 25397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.226279 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "949c2e7084c948a5baf753a42377e6ea"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "9fb8edd7f209d8c82ad9201ba5c3fe09"
server_key_iv: "2673d5b684954cfccf99e9d5c1c574f6"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.226356 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.257704 25397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.260329 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.260517 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.260874 25397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:26.261246 25397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:26.261307 25397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.261351 25397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:26.261376 25397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.267622 25397 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:46233
I20260927 21:02:26.267699 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:46233 every 8 connection(s)
I20260927 21:02:26.268061 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:26.269555 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25397
I20260927 21:02:26.269636 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:26.269860 17147 external_mini_cluster.cc:1428] Setting key b592c7fdd823f2e200f30a318fe9d423
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:26.278779 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.268912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:58351 (local address 127.16.190.254:39099)
0927 21:02:26.268962 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.268964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.269193 (+ 229us) server_negotiation.cc:407] Connection header received
0927 21:02:26.269734 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.269735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.269748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.269822 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.270937 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.271745 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.272790 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.273148 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.273219 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.275413 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:26.275415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:26.275415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:26.275423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:26.277220 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.277653 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.277656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.277657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.277721 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.278014 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.278015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.278015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.278183 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:26.278259 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:26.278561 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:02:26.278665 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:02:26.278951 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.268890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:58351)
0927 21:02:26.269055 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.269090 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.269580 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.269880 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.269884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.270618 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.270844 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.270849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.271851 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.271852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.272694 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.272696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.273243 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.273243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.273350 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.273908 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.273916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.275314 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:26.277318 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.277324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.277327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.277568 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.277814 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.277815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.277815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.277930 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.278271 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:26.278276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:26.278409 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 21:02:26.278502 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:02:26.278707 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 21:02:26.279466 25526 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39099
I20260927 21:02:26.279561 25526 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:26.279884 25526 heartbeater.cc:511] Master 127.16.190.254:39099 requested a full tablet report, sending...
I20260927 21:02:26.280545 25335 ts_manager.cc:194] Registered new tserver with Master: 949c2e7084c948a5baf753a42377e6ea (127.16.190.193:46233)
I20260927 21:02:26.281291 25335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:58351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:26.300285 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.382635 25531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.382864 25531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.382889 25531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.384099 25531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.384152 25531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.384182 25531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:26.385751 25531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.386016 25531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.386258 25531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.391299 25536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.391327 25539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.391321 25537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.391575 25531 server_base.cc:1034] running on GCE node
I20260927 21:02:26.391862 25531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.392110 25531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.393267 25531 hybrid_clock.cc:648] HybridClock initialized: now 1790542946393248 us; error 39 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.395881 25531 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:26.396332 25531 webserver.cc:460] Webserver started at http://127.16.190.194:38579/ using document root <none> and password file <none>
I20260927 21:02:26.396529 25531 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.396569 25531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.396660 25531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:26.397771 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "ac287aa0ba5f44378bb58940c92839ed"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "3e036b163a093308566b6a84294efb70"
server_key_iv: "7a05036b76b25612524c59022acc9092"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.398066 25531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "ac287aa0ba5f44378bb58940c92839ed"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "3e036b163a093308566b6a84294efb70"
server_key_iv: "7a05036b76b25612524c59022acc9092"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.399307 25531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:26.400045 25546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.400255 25531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.400372 25531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "ac287aa0ba5f44378bb58940c92839ed"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "3e036b163a093308566b6a84294efb70"
server_key_iv: "7a05036b76b25612524c59022acc9092"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.400449 25531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.428725 25531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.431386 25531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.431541 25531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.431804 25531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:26.432199 25531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:26.432235 25531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.432260 25531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:26.432272 25531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.438277 25531 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44513
I20260927 21:02:26.438350 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44513 every 8 connection(s)
I20260927 21:02:26.438715 25531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:26.445739 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25531
I20260927 21:02:26.445859 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:26.446089 17147 external_mini_cluster.cc:1428] Setting key 1429413c102319227c4140ae0364d15a
I20260927 21:02:26.449239 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.439543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51037 (local address 127.16.190.254:39099)
0927 21:02:26.439617 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.439620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.439834 (+ 214us) server_negotiation.cc:407] Connection header received
0927 21:02:26.440347 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.440348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.440360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.440417 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.441456 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.441976 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.443119 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.443310 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.443370 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.445811 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:26.445813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:26.445814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:26.445823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:26.447483 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.448026 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.448027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.448027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.448101 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.448436 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.448437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.448438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.448563 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:26.448670 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:26.449100 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 21:02:26.449172 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:02:26.449658 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.439513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:51037)
0927 21:02:26.439699 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.439738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.440246 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.440490 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.440494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.441149 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.441353 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.441358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.442089 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.442090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.443009 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.443011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.443392 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.443393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.443510 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.444069 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.444078 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.445689 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:26.447597 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.447605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.447608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.447935 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.448175 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.448176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.448177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.448341 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.448684 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:26.448691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:26.448913 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 21:02:26.449056 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 21:02:26.449259 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260927 21:02:26.450237 25660 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39099
I20260927 21:02:26.450353 25660 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:26.450587 25660 heartbeater.cc:511] Master 127.16.190.254:39099 requested a full tablet report, sending...
I20260927 21:02:26.450987 25335 ts_manager.cc:194] Registered new tserver with Master: ac287aa0ba5f44378bb58940c92839ed (127.16.190.194:44513)
I20260927 21:02:26.451382 25335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:51037
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:26.475646 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.560357 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.560580 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.560606 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.561853 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.561897 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.561924 25665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:26.563382 25665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.563647 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.563885 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.569373 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.569456 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.569396 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.569535 25665 server_base.cc:1034] running on GCE node
I20260927 21:02:26.569806 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.570057 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.571269 25665 hybrid_clock.cc:648] HybridClock initialized: now 1790542946571247 us; error 36 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.573891 25665 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:26.574407 25665 webserver.cc:460] Webserver started at http://127.16.190.195:33437/ using document root <none> and password file <none>
I20260927 21:02:26.574640 25665 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.574712 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.574867 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:26.575968 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "acaf6bfd1e5544b2a2abf07a8992f528"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "4c18fa0a3e0d77e8363f514dbddc5ec2"
server_key_iv: "a7b2d950ca856df9507e86a25bc8add9"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.576362 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "acaf6bfd1e5544b2a2abf07a8992f528"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "4c18fa0a3e0d77e8363f514dbddc5ec2"
server_key_iv: "a7b2d950ca856df9507e86a25bc8add9"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.577906 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:26.578755 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.579139 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.579282 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "acaf6bfd1e5544b2a2abf07a8992f528"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "4c18fa0a3e0d77e8363f514dbddc5ec2"
server_key_iv: "a7b2d950ca856df9507e86a25bc8add9"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.579370 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.590531 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.593123 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.593304 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.593559 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:26.593887 25665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:26.593945 25665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.593989 25665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:26.594015 25665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.600461 25665 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:41479
I20260927 21:02:26.600610 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:41479 every 8 connection(s)
I20260927 21:02:26.600944 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:26.610406 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.601729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:47181 (local address 127.16.190.254:39099)
0927 21:02:26.601800 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.601802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.602037 (+ 235us) server_negotiation.cc:407] Connection header received
0927 21:02:26.602510 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.602511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.602520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.602581 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.603642 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.604155 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.605288 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.605486 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.605552 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.607629 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:26.607630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:26.607630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:26.607637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:26.609067 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.609461 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.609462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.609463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.609553 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:26.609783 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:26.609783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:26.609784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:26.609892 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:26.609949 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:26.610263 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 21:02:26.610325 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:02:26.610591 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.601710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:47181)
0927 21:02:26.601889 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.601931 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.602426 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.602634 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.602636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.603347 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.603553 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.603557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.604316 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.604317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.605212 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.605214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.605600 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.605601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.605713 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.606181 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.606189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.607546 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:26.609154 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.609159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.609161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.609387 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.609617 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:26.609618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:26.609618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:26.609720 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:26.609982 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:26.609988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:26.610123 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 21:02:26.610224 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 21:02:26.610359 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260927 21:02:26.611070 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25665
I20260927 21:02:26.611167 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:26.611255 25794 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39099
I20260927 21:02:26.611353 17147 external_mini_cluster.cc:1428] Setting key 6632d02014275dc21c157b6797f674e8
I20260927 21:02:26.611379 25794 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:26.611584 25794 heartbeater.cc:511] Master 127.16.190.254:39099 requested a full tablet report, sending...
I20260927 21:02:26.611994 25335 ts_manager.cc:194] Registered new tserver with Master: acaf6bfd1e5544b2a2abf07a8992f528 (127.16.190.195:41479)
I20260927 21:02:26.612321 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:02:26.612370 25335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:47181
WARNING: no policy specified for oryx/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
Entry for principal oryx/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab.
WARNING: no policy specified for oryx/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
Entry for principal oryx/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab.
WARNING: no policy specified for oryx/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
Entry for principal oryx/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab.
I20260927 21:02:26.692500 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25397
I20260927 21:02:26.697081 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25531
I20260927 21:02:26.701552 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25665
I20260927 21:02:26.707108 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25303
I20260927 21:02:26.712271 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39099
--webserver_interface=127.16.190.254
--webserver_port=33705
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.792888 25803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.793115 25803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.793138 25803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.794370 25803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:26.794420 25803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.794433 25803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.794446 25803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:26.794458 25803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:26.795974 25803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=33705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.796247 25803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.796494 25803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.801496 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.801496 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.801518 25811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.801854 25803 server_base.cc:1034] running on GCE node
I20260927 21:02:26.802162 25803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.802451 25803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.803639 25803 hybrid_clock.cc:648] HybridClock initialized: now 1790542946803612 us; error 53 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.806237 25803 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:26.806767 25803 webserver.cc:460] Webserver started at http://127.16.190.254:33705/ using document root <none> and password file <none>
I20260927 21:02:26.807036 25803 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.807127 25803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.808564 25803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.809314 25818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.809538 25803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:26.809646 25803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "9d59d94683e34ecab75796a0ad5105d6"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "e8de0d1cd15eea285c84897c5a8b2be4"
server_key_iv: "b4c3ae416977af36b287e878704747fe"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.810016 25803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.821525 25803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.823783 25803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.823959 25803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.828049 25803 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39099
I20260927 21:02:26.828097 25870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39099 every 8 connection(s)
I20260927 21:02:26.828506 25803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:26.830045 25871 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:02:26.830651 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Bootstrap starting.
I20260927 21:02:26.837249 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25803
I20260927 21:02:26.837639 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:46233
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:39099
--webserver_port=38335
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:02:26.839041 25871 log.cc:824] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:26.841049 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:02:26.841351 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Bootstrap complete.
I20260927 21:02:26.843529 25871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.843751 25871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d59d94683e34ecab75796a0ad5105d6, State: Initialized, Role: FOLLOWER
I20260927 21:02:26.843915 25871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.843983 25871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:26.844043 25871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:26.844090 25871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:02:26.845165 25871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.845252 25871 leader_election.cc:302] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d59d94683e34ecab75796a0ad5105d6; no voters:
I20260927 21:02:26.845510 25871 leader_election.cc:288] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:02:26.845561 25875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:02:26.845685 25875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [term 2 LEADER]: Becoming Leader. State: Replica: 9d59d94683e34ecab75796a0ad5105d6, State: Running, Role: LEADER
I20260927 21:02:26.845786 25871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:26.845768 25875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } }
I20260927 21:02:26.846194 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d59d94683e34ecab75796a0ad5105d6. Latest consensus state: current_term: 2 leader_uuid: "9d59d94683e34ecab75796a0ad5105d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } } }
I20260927 21:02:26.846299 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:26.846203 25876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d59d94683e34ecab75796a0ad5105d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d59d94683e34ecab75796a0ad5105d6" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39099 } } }
I20260927 21:02:26.846501 25876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:26.847052 25883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:26.847613 25883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:26.847739 25883 catalog_manager.cc:1259] Loaded cluster ID: 33bb7cd9d2054fde945a008842327ba4
I20260927 21:02:26.847783 25883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:26.848950 25883 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:26.849133 25883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d59d94683e34ecab75796a0ad5105d6: Loaded TSK: 0
I20260927 21:02:26.849352 25883 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:02:26.921478 25873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:26.921734 25873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:26.921777 25873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:26.923031 25873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:26.923101 25873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:26.923156 25873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:26.924702 25873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.193.keytab
--principal=oryx/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:46233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=38335
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:26.925017 25873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:26.925266 25873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:26.930512 25898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.930588 25900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:26.930512 25897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:26.930830 25873 server_base.cc:1034] running on GCE node
I20260927 21:02:26.931095 25873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:26.931398 25873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:26.932579 25873 hybrid_clock.cc:648] HybridClock initialized: now 1790542946932550 us; error 42 us; skew 500 ppm
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542946, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:26.935590 25873 init.cc:377] Logged in from keytab as oryx/127.16.190.193@KRBTEST.COM (short username oryx)
I20260927 21:02:26.936057 25873 webserver.cc:460] Webserver started at http://127.16.190.193:38335/ using document root <none> and password file <none>
I20260927 21:02:26.936298 25873 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:26.936384 25873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:26.937937 25873 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.938694 25907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.938917 25873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:26.939041 25873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "949c2e7084c948a5baf753a42377e6ea"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "9fb8edd7f209d8c82ad9201ba5c3fe09"
server_key_iv: "2673d5b684954cfccf99e9d5c1c574f6"
server_key_version: "encryptionkey@0"
I20260927 21:02:26.939421 25873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:26.953271 25873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:26.955802 25873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:26.955996 25873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:26.956243 25873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:26.956564 25873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:26.956620 25873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.956665 25873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:26.956687 25873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:26.963183 25873 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:46233
I20260927 21:02:26.963222 26020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:46233 every 8 connection(s)
I20260927 21:02:26.963593 25873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:26.973284 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 25873
I20260927 21:02:26.973625 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.964388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:45155)
0927 21:02:26.964564 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.964607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.965181 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.965665 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.965668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.966276 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.966475 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.966480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.968190 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.968191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.969138 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.969141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.970046 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.970047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.970156 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.970812 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.970821 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.973416 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260927 21:02:26.973759 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:44513
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:39099
--webserver_port=38579
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:26.973858 26021 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:39099 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.190.254:39099: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
W20260927 21:02:26.974406 26024 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:26.964464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:45155 (local address 127.16.190.254:39099)
0927 21:02:26.964666 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.964675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.964714 (+ 39us) server_negotiation.cc:407] Connection header received
0927 21:02:26.965403 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.965410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.965467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.965620 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.966606 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.968072 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.969234 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.969926 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.970043 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.974191 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:45155: BlockingRecv error: recv got EOF from 127.16.190.193:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:26.979907 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.974050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:56333)
0927 21:02:26.974351 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.974361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.974401 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.974642 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.974643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.974792 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.974851 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.974853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.975802 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.975803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.976464 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.976467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.976927 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.976928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.977004 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.977500 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.977505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.979800 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":75,"threads_started":1}
W20260927 21:02:26.979990 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:26.974224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:56333 (local address 127.16.190.254:39099)
0927 21:02:26.974521 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.974522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.974529 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:26.974550 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.974551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.974560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.974914 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.974945 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.975710 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.976581 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.976780 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.976927 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.979903 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:56333: BlockingRecv error: recv got EOF from 127.16.190.193:56333 (error 108)
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":74,"threads_started":1}
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:26 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:26.985743 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:26.980522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:38513)
0927 21:02:26.980633 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:26.980644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:26.980728 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:26.980926 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:26.980927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:26.981077 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:26.981112 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.981114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.981853 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.981854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.982409 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:26.982410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.982833 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.982834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:26.982893 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.983420 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:26.983428 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:26.985638 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:02:26.985792 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:26.980571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:38513 (local address 127.16.190.254:39099)
0927 21:02:26.980614 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:26.980617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:26.980711 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:02:26.980834 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:26.980835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:26.980849 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:26.980911 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:26.981211 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.981759 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.982516 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:26.982751 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:26.982839 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:26.985729 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:38513: BlockingRecv error: recv got EOF from 127.16.190.193:38513 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:02:26.985920 26021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:02:27.058182 26027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:27.058403 26027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:27.058429 26027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:27.059664 26027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:27.059712 26027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:27.059739 26027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:27.061236 26027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.194.keytab
--principal=oryx/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:44513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=38579
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:27.061508 26027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:27.061746 26027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:27.067054 26035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:27.067116 26027 server_base.cc:1034] running on GCE node
W20260927 21:02:27.067061 26033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:27.067063 26032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:27.067574 26027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:27.067782 26027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:27.068876 26027 hybrid_clock.cc:648] HybridClock initialized: now 1790542947068861 us; error 29 us; skew 500 ppm
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542947, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:27.071728 26027 init.cc:377] Logged in from keytab as oryx/127.16.190.194@KRBTEST.COM (short username oryx)
I20260927 21:02:27.072224 26027 webserver.cc:460] Webserver started at http://127.16.190.194:38579/ using document root <none> and password file <none>
I20260927 21:02:27.072474 26027 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:27.072532 26027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:27.074048 26027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:27.074697 26042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.074906 26027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:27.074999 26027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "ac287aa0ba5f44378bb58940c92839ed"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "3e036b163a093308566b6a84294efb70"
server_key_iv: "7a05036b76b25612524c59022acc9092"
server_key_version: "encryptionkey@0"
I20260927 21:02:27.075350 26027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:27.086372 26027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:27.089303 26027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:27.089459 26027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:27.089708 26027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:27.090030 26027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:27.090062 26027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.090086 26027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:27.090132 26027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.095817 26027 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44513
I20260927 21:02:27.096028 26155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44513 every 8 connection(s)
I20260927 21:02:27.096316 26027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:27.099507 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26027
I20260927 21:02:27.099843 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:41479
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:39099
--webserver_port=33437
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 21:02:27.104794 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.097156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43939 (local address 127.16.190.254:39099)
0927 21:02:27.097226 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.097228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.097458 (+ 230us) server_negotiation.cc:407] Connection header received
0927 21:02:27.097947 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.097948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.097960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.098021 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.098957 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.099445 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.101092 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.101451 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.101532 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.104734 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:43939: BlockingRecv error: recv got EOF from 127.16.190.194:43939 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:02:27.104890 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.097145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:43939)
0927 21:02:27.097327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.097370 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.097863 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.098079 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.098082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.098671 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.098872 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.098877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.099562 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.099564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.100978 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.100982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.101602 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.101603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.101758 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.102377 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.102386 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.104650 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
W20260927 21:02:27.105099 26156 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:39099 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.190.254:39099: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.110219 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.105334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:54693)
0927 21:02:27.105385 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.105398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.105444 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.105624 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.105625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.105738 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.105785 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.105786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.106786 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.106786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.107284 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.107286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.107793 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.107794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.107844 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.108350 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.108364 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.110168 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:02:27.110342 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.105424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:54693 (local address 127.16.190.254:39099)
0927 21:02:27.105490 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.105492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.105501 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:27.105529 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.105530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.105547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.105621 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.105858 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.106663 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.107371 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.107715 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.107791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.110280 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:54693: BlockingRecv error: recv got EOF from 127.16.190.194:54693 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.114909 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.110653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:41217)
0927 21:02:27.110706 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.110716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.110760 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.110909 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.110909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.111035 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.111062 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.111063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.111685 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.111686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.112258 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.112260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.112682 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.112683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.112736 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.113231 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.113235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.114825 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:02:27.114962 26026 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.110719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:41217 (local address 127.16.190.254:39099)
0927 21:02:27.110782 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.110784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.110792 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:27.110828 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.110828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.110837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.110905 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.111144 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.111597 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.112349 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.112592 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.112681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.114902 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:41217: BlockingRecv error: recv got EOF from 127.16.190.194:41217 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 21:02:27.115041 26156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:02:27.185127 26159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:27.185392 26159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:27.185433 26159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:27.186668 26159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:27.186746 26159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:27.186792 26159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:27.188295 26159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/krb5kdc/oryx_127.16.190.195.keytab
--principal=oryx/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:41479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=33437
--tserver_master_addrs=127.16.190.254:39099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:27.188573 26159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:27.188829 26159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:02:27.194314 26159 server_base.cc:1034] running on GCE node
W20260927 21:02:27.194286 26164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:27.194279 26165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:27.194279 26167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:27.194777 26159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:27.195003 26159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:27.196209 26159 hybrid_clock.cc:648] HybridClock initialized: now 1790542947196195 us; error 38 us; skew 500 ppm
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542947, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:27.198956 26159 init.cc:377] Logged in from keytab as oryx/127.16.190.195@KRBTEST.COM (short username oryx)
I20260927 21:02:27.199416 26159 webserver.cc:460] Webserver started at http://127.16.190.195:33437/ using document root <none> and password file <none>
I20260927 21:02:27.199662 26159 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:27.199749 26159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:27.201145 26159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:02:27.201722 26174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.201893 26159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:02:27.202003 26159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "acaf6bfd1e5544b2a2abf07a8992f528"
format_stamp: "Formatted at 2026-09-27 21:02:26 on dist-test-slave-6c7s"
server_key: "4c18fa0a3e0d77e8363f514dbddc5ec2"
server_key_iv: "a7b2d950ca856df9507e86a25bc8add9"
server_key_version: "encryptionkey@0"
I20260927 21:02:27.202368 26159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:27.225775 26159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:27.228086 26159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:27.228291 26159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:27.228543 26159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:27.228925 26159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:27.228978 26159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.229015 26159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:27.229043 26159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:27.234781 26159 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:41479
I20260927 21:02:27.234858 26287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:41479 every 8 connection(s)
I20260927 21:02:27.235225 26159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:27.235505 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26159
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.243500 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.236046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.16.190.254:39099)
0927 21:02:27.236100 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.236102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.236300 (+ 198us) server_negotiation.cc:407] Connection header received
0927 21:02:27.236392 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.236393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.236402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.236453 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.237138 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.237654 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.238457 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.238668 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.238744 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.240017 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:27.240022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:27.240022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:27.240038 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:27.241917 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:27.242334 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:27.242338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:27.242338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:27.242473 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:27.242729 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:27.242731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:27.242731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:27.243015 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:27.243128 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:27.243308 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 21:02:27.243398 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 21:02:27.243692 26024 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.236170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:45749 (local address 127.16.190.254:39099)
0927 21:02:27.236205 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.236206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.236387 (+ 181us) server_negotiation.cc:407] Connection header received
0927 21:02:27.236958 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.236959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.236976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.237023 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.238006 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.238487 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.239717 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.239912 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.240016 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.243660 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:45749: BlockingRecv error: recv got EOF from 127.16.190.195:45749 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:02:27.243779 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.236078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:45749)
0927 21:02:27.236242 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.236311 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.236856 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.237130 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.237134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.237715 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.237908 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.237913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.238604 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.238604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.239609 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.239611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.240025 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.240026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.240159 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.240749 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.240757 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.243541 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260927 21:02:27.244005 26288 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:39099 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.190.254:39099: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.249207 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.244227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:44243)
0927 21:02:27.244298 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.244308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.244340 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.244671 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.244672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.244777 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.244893 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.244895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.245538 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.245539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.246263 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.246267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.246697 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.246698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.246811 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.247316 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.247321 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.249154 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 21:02:27.249356 26024 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.244526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:44243 (local address 127.16.190.254:39099)
0927 21:02:27.244570 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.244571 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.244577 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:02:27.244591 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.244592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.244600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.244643 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.244950 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.245399 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.246349 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.246572 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.246639 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.249270 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:44243: BlockingRecv error: recv got EOF from 127.16.190.195:44243 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.253432 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.249544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:51299)
0927 21:02:27.249598 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.249609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.249655 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.249796 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.249796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.249918 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.249948 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.249949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.250543 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.250544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.251034 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.251036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.251349 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.251350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.251386 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.251766 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.251770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.253380 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 21:02:27.253583 26288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:02:27.253607 26024 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.249608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:51299 (local address 127.16.190.254:39099)
0927 21:02:27.249646 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.249648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.249657 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:27.249714 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.249715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.249724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.249774 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.250010 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.250479 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.251107 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.251290 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.251339 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.253537 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:51299: BlockingRecv error: recv got EOF from 127.16.190.195:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:27 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:27.991267 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:27.986409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:37623)
0927 21:02:27.986630 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:27.986644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:27.986715 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:27.986873 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:27.986874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:27.987037 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:27.987096 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.987100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.987834 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.987835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.988388 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:27.988390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.988715 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.988715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:27.988759 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.989318 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:27.989323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:27.991196 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260927 21:02:27.991400 26293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:27.986524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:37623 (local address 127.16.190.254:39099)
0927 21:02:27.986674 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:27.986676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:27.986699 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:02:27.986776 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:27.986776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:27.986795 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:27.986852 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:27.987195 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.987756 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.988463 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:27.988649 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:27.988699 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:27.991307 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:37623: BlockingRecv error: recv got EOF from 127.16.190.193:37623 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:28.120376 26294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:28.115453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:47321)
0927 21:02:28.115639 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:28.115650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:28.115791 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:28.115957 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:28.115957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:28.116178 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:28.116243 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.116245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.116958 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.116960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.117678 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.117682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.118012 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.118013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.118053 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.118483 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:28.118488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:28.120309 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
W20260927 21:02:28.120525 26293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:28.115535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:47321 (local address 127.16.190.254:39099)
0927 21:02:28.115614 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:28.115617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:28.115768 (+ 151us) server_negotiation.cc:407] Connection header received
0927 21:02:28.115858 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:28.115859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:28.115874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:28.115951 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:28.116314 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.116860 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.117755 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.117942 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.117996 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.120455 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:47321: BlockingRecv error: recv got EOF from 127.16.190.194:47321 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:28.258775 26295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:28.254022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:53053)
0927 21:02:28.254171 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:28.254181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:28.254226 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:28.254382 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:28.254383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:28.254543 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:28.254593 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.254595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.255318 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.255319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.255873 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.255875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.256256 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.256257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.256299 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.256757 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:28.256762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:28.258695 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260927 21:02:28.258930 26293 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:28.254171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:53053 (local address 127.16.190.254:39099)
0927 21:02:28.254244 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:28.254246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:28.254261 (+ 15us) server_negotiation.cc:407] Connection header received
0927 21:02:28.254284 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:28.254285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:28.254296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:28.254356 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:28.254691 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.255197 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.255972 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.256180 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.256249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.258832 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:53053: BlockingRecv error: recv got EOF from 127.16.190.195:53053 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:28 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:28.996533 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:28.991822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:46581)
0927 21:02:28.991970 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:28.991980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:28.992030 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:28.992258 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:28.992259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:28.992413 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:28.992466 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.992467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.993219 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.993220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.993744 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:28.993746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.994115 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.994116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:28.994154 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.994608 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:28.994612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:28.996434 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260927 21:02:28.996600 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:28.991946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:46581 (local address 127.16.190.254:39099)
0927 21:02:28.992105 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:28.992108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:28.992118 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:28.992146 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:28.992147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:28.992160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:28.992247 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:28.992538 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.993149 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.993832 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:28.994033 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:28.994108 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:28.996518 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:46581: BlockingRecv error: recv got EOF from 127.16.190.193:46581 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:29.126268 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:29.121041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:59085)
0927 21:02:29.121263 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:29.121282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:29.121337 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:29.121601 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:29.121602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:29.121777 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:29.121828 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.121830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.122914 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.122915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.123456 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.123458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.123842 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.123843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.123882 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:29.124324 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:29.124328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:29.126170 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":126,"threads_started":1}
W20260927 21:02:29.126318 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:29.121200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:59085 (local address 127.16.190.254:39099)
0927 21:02:29.121318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:29.121320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:29.121333 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:02:29.121403 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:29.121404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:29.121415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:29.121594 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:29.121932 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.122733 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.123582 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.123772 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.123831 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:29.126255 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:59085: BlockingRecv error: recv got EOF from 127.16.190.194:59085 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:29.264219 26299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:29.259390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:36797)
0927 21:02:29.259622 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:29.259634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:29.259728 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:29.259900 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:29.259901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:29.260088 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:29.260135 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.260137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.260798 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.260799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.261372 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.261373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.261746 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.261747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.261809 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:29.262353 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:29.262358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:29.264120 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
W20260927 21:02:29.264271 26297 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:29.259533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:36797 (local address 127.16.190.254:39099)
0927 21:02:29.259606 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:29.259609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:29.259711 (+ 102us) server_negotiation.cc:407] Connection header received
0927 21:02:29.259787 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:29.259788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:29.259801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:29.259879 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:29.260219 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.260713 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.261466 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.261667 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.261748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:29.264203 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:36797: BlockingRecv error: recv got EOF from 127.16.190.195:36797 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:29 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:30.002086 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:29.997186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:51621)
0927 21:02:29.997338 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:29.997351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:29.997432 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:29.997613 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:29.997614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:29.997784 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:29.997836 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.997838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.998506 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.998507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.999041 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:29.999043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.999414 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.999414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:29.999454 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:29.999953 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:29.999958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:30.001984 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
W20260927 21:02:30.002138 26301 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:29.997275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:51621 (local address 127.16.190.254:39099)
0927 21:02:29.997438 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:29.997440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:29.997450 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:29.997487 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:29.997488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:29.997497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:29.997595 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:29.997901 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.998402 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.999136 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:29.999338 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:29.999413 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:30.002064 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:51621: BlockingRecv error: recv got EOF from 127.16.190.193:51621 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
Sep 27 21:02:30 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:30 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:30.131582 26302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:30.126852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:59261)
0927 21:02:30.127024 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:30.127036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:30.127104 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:30.127285 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:30.127286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:30.127451 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:30.127501 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:30.127502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.128194 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.128195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:30.128717 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:30.128718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.129075 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.129075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:30.129120 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:30.129710 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:30.129714 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:30.131491 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
W20260927 21:02:30.131722 26301 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:30.126976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:59261 (local address 127.16.190.254:39099)
0927 21:02:30.127039 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:30.127045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:30.127092 (+ 47us) server_negotiation.cc:407] Connection header received
0927 21:02:30.127177 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:30.127178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:30.127190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:30.127261 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:30.127588 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.128109 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.128791 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.129003 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.129057 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:30.131649 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:59261: BlockingRecv error: recv got EOF from 127.16.190.194:59261 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 21:02:30 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:30 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:30.269879 26303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:30.264967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:44039)
0927 21:02:30.265136 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:30.265147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:30.265235 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:30.265391 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:30.265392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:30.265621 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:30.265667 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:30.265670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.266355 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.266357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:30.266893 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:30.266894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.267269 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.267270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:30.267330 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:30.267768 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:30.267773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:30.269780 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
W20260927 21:02:30.269932 26301 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:30.265003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:44039 (local address 127.16.190.254:39099)
0927 21:02:30.265080 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:30.265083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:30.265216 (+ 133us) server_negotiation.cc:407] Connection header received
0927 21:02:30.265298 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:30.265299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:30.265313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:30.265382 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:30.265756 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.266268 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.266981 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:30.267182 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:30.267269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:30.269868 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:44039: BlockingRecv error: recv got EOF from 127.16.190.195:44039 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:31.007900 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:31.002758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:60903)
0927 21:02:31.002908 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:31.002925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:31.003027 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:31.003229 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:31.003230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:31.003400 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:31.003451 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.003452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.004155 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.004156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.004685 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.004686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.005272 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.005273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.005349 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.005811 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:31.005816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:31.007802 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
W20260927 21:02:31.007977 26305 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:31.002879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:60903 (local address 127.16.190.254:39099)
0927 21:02:31.003028 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:31.003030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:31.003042 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:31.003089 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:31.003089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:31.003099 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:31.003177 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:31.003567 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.004074 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.004795 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.005161 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.005270 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.007900 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:60903: BlockingRecv error: recv got EOF from 127.16.190.193:60903 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:31.141104 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:31.132100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:48771)
0927 21:02:31.132247 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:31.132258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:31.132319 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:31.132447 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:31.132448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:31.132601 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:31.132651 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.132652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.133422 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.133423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.133945 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.133947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.134299 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.134299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.134337 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.139052 (+ 4715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:31.139058 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:31.141033 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
W20260927 21:02:31.141294 26305 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:31.132184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:48771 (local address 127.16.190.254:39099)
0927 21:02:31.132235 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:31.132237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:31.132306 (+ 69us) server_negotiation.cc:407] Connection header received
0927 21:02:31.132374 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:31.132375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:31.132386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:31.132430 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:31.132723 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.133349 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.134023 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.134233 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.134289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.141205 (+ 6916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:48771: BlockingRecv error: recv got EOF from 127.16.190.194:48771 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:31 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:31.275467 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:31.270465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:42977)
0927 21:02:31.270726 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:31.270741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:31.270842 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:31.271008 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:31.271009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:31.271195 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:31.271246 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.271248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.271974 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.271976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.272517 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:31.272519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.272937 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.272939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:31.273002 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.273450 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:31.273456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:31.275365 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":143,"threads_started":1}
W20260927 21:02:31.275516 26305 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:31.270624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:42977 (local address 127.16.190.254:39099)
0927 21:02:31.270728 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:31.270730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:31.270820 (+ 90us) server_negotiation.cc:407] Connection header received
0927 21:02:31.270912 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:31.270913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:31.270932 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:31.271000 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:31.271330 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.271873 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.272605 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:31.272805 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:31.272928 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:31.275449 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:42977: BlockingRecv error: recv got EOF from 127.16.190.195:42977 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:32.013451 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:32.008483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:50813)
0927 21:02:32.008675 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:32.008686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:32.008736 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:32.008927 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:32.008928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:32.009093 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:32.009145 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.009147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.009825 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.009826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.010357 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.010359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.010707 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.010708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.010749 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.011184 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:32.011189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:32.013390 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
W20260927 21:02:32.013659 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:32.008562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:50813 (local address 127.16.190.254:39099)
0927 21:02:32.008725 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:32.008727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:32.008736 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:32.008783 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:32.008783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:32.008793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:32.008911 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:32.009213 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.009742 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.010432 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.010624 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.010694 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.013520 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:50813: BlockingRecv error: recv got EOF from 127.16.190.193:50813 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:32.146695 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:32.141650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:40739)
0927 21:02:32.141822 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:32.141836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:32.141912 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:32.142078 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:32.142079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:32.142252 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:32.142312 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.142314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.143157 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.143159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.143719 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.143721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.144059 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.144060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.144119 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.144604 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:32.144609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:32.146558 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
W20260927 21:02:32.146706 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:32.141835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:40739 (local address 127.16.190.254:39099)
0927 21:02:32.141901 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:32.141903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:32.141911 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:32.142000 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:32.142001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:32.142010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:32.142061 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:32.142392 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.143011 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.143794 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.143972 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.144025 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.146659 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:40739: BlockingRecv error: recv got EOF from 127.16.190.194:40739 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:32 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:32.280923 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:32.276111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:40831)
0927 21:02:32.276258 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:32.276272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:32.276343 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:32.276504 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:32.276505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:32.276693 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:32.276743 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.276745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.277520 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.277522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.278121 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:32.278123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.278491 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.278492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:32.278534 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.279033 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:32.279039 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:32.280803 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260927 21:02:32.281040 26309 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:32.276226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:40831 (local address 127.16.190.254:39099)
0927 21:02:32.276296 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:32.276298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:32.276327 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:02:32.276410 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:32.276411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:32.276424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:32.276481 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:32.276876 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.277372 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.278223 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:32.278421 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:32.278479 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:32.280957 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:40831: BlockingRecv error: recv got EOF from 127.16.190.195:40831 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:33.019001 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:33.014125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:58873)
0927 21:02:33.014246 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:33.014258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:33.014414 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:33.014591 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:33.014592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:33.014812 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:33.014861 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.014862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.015549 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.015550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.016091 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.016092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.016465 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.016466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.016523 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.017087 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:33.017091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:33.018905 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
W20260927 21:02:33.019073 26313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:33.014160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:58873 (local address 127.16.190.254:39099)
0927 21:02:33.014305 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:33.014308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:33.014345 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:02:33.014499 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:33.014500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:33.014510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:33.014582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:33.014947 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.015462 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.016185 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.016389 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.016466 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.019000 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:58873: BlockingRecv error: recv got EOF from 127.16.190.193:58873 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:33.152333 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:33.147321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:38457)
0927 21:02:33.147462 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:33.147471 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:33.147573 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:33.147735 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:33.147736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:33.147922 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:33.147971 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.147972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.148679 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.148680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.149321 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.149324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.149745 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.149746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.149807 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.150328 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:33.150333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:33.152271 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260927 21:02:33.152491 26313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:33.147362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:38457 (local address 127.16.190.254:39099)
0927 21:02:33.147411 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:33.147413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:33.147545 (+ 132us) server_negotiation.cc:407] Connection header received
0927 21:02:33.147636 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:33.147637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:33.147655 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:33.147729 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:33.148067 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.148581 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.149411 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.149651 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.149744 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.152412 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:38457: BlockingRecv error: recv got EOF from 127.16.190.194:38457 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:33 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:33.286526 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:33.281491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:58525)
0927 21:02:33.281691 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:33.281703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:33.281777 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:33.281893 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:33.281893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:33.282074 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:33.282124 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.282126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.282733 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.282734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.283331 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:33.283332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.283669 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.283670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:33.283721 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.284360 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:33.284365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:33.286447 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
W20260927 21:02:33.286597 26313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:33.281587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:58525 (local address 127.16.190.254:39099)
0927 21:02:33.281632 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:33.281634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:33.281765 (+ 131us) server_negotiation.cc:407] Connection header received
0927 21:02:33.281831 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:33.281832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:33.281845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:33.281879 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:33.282192 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.282675 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.283406 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:33.283585 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:33.283636 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:33.286546 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:58525: BlockingRecv error: recv got EOF from 127.16.190.195:58525 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:34.025887 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:34.019537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:46343)
0927 21:02:34.019673 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:34.019682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:34.019727 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:34.019945 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:34.019946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:34.020105 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:34.020199 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.020201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.021306 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.021308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.022283 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.022297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.022819 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.022820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.022878 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.023428 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:34.023434 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:34.025801 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
W20260927 21:02:34.026001 26317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:34.019663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:46343 (local address 127.16.190.254:39099)
0927 21:02:34.019808 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:34.019809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:34.019818 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:34.019843 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:34.019844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:34.019854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:34.019924 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:34.020307 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.021194 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.022377 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.022719 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.022782 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.025929 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:46343: BlockingRecv error: recv got EOF from 127.16.190.193:46343 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:34.157599 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:34.152846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:55895)
0927 21:02:34.152987 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:34.152998 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:34.153089 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:34.153276 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:34.153277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:34.153460 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:34.153508 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.153510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.154204 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.154205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.154759 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.154761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.155138 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.155139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.155198 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.155704 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:34.155710 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:34.157502 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
W20260927 21:02:34.157660 26317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:34.152906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:55895 (local address 127.16.190.254:39099)
0927 21:02:34.152962 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:34.152964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:34.153067 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:02:34.153179 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:34.153180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:34.153194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:34.153267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:34.153589 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.154110 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.154856 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.155059 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.155136 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.157588 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:55895: BlockingRecv error: recv got EOF from 127.16.190.194:55895 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:34 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:34.291581 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:34.287070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:49435)
0927 21:02:34.287261 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:34.287272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:34.287335 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:34.287474 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:34.287474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:34.287632 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:34.287689 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.287691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.288322 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.288323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.288880 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:34.288882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.289214 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.289215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:34.289254 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.289725 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:34.289730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:34.291485 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
W20260927 21:02:34.291648 26317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:34.287134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:49435 (local address 127.16.190.254:39099)
0927 21:02:34.287222 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:34.287224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:34.287335 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:02:34.287392 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:34.287393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:34.287404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:34.287446 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:34.287753 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.288244 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.288960 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:34.289146 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:34.289201 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:34.291569 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:49435: BlockingRecv error: recv got EOF from 127.16.190.195:49435 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:35.031673 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:35.026728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:39919)
0927 21:02:35.026897 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:35.026911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:35.026978 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:35.027213 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:35.027214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:35.027377 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:35.027428 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.027429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.028162 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.028163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.028707 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.028709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.029149 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.029150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.029197 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.029697 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:35.029701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:35.031599 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
W20260927 21:02:35.031757 26321 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:35.026872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:39919 (local address 127.16.190.254:39099)
0927 21:02:35.027074 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:35.027076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:35.027085 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:35.027109 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:35.027110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:35.027121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:35.027192 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:35.027496 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.028070 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.028789 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.029074 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.029134 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.031699 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:39919: BlockingRecv error: recv got EOF from 127.16.190.193:39919 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:35.163116 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:35.158128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:52841)
0927 21:02:35.158276 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:35.158291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:35.158349 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:35.158518 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:35.158519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:35.158682 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:35.158730 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.158732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.159517 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.159518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.160045 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.160046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.160401 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.160402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.160466 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.161094 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:35.161099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:35.163005 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260927 21:02:35.163179 26321 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:35.158190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:52841 (local address 127.16.190.254:39099)
0927 21:02:35.158251 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:35.158255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:35.158347 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:02:35.158408 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:35.158408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:35.158420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:35.158462 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:35.158798 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.159437 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.160116 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.160317 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.160367 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.163114 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:52841: BlockingRecv error: recv got EOF from 127.16.190.194:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:35 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:35.297407 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:35.292176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:56807)
0927 21:02:35.292357 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:35.292370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:35.292437 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:35.292569 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:35.292570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:35.292738 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:35.292787 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.292789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.293604 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.293605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.294141 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:35.294143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.294590 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.294591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:35.294644 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.295130 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:35.295134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:35.297323 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
W20260927 21:02:35.297499 26321 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:35.292297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:56807 (local address 127.16.190.254:39099)
0927 21:02:35.292347 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:35.292349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:35.292444 (+ 95us) server_negotiation.cc:407] Connection header received
0927 21:02:35.292488 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:35.292489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:35.292502 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:35.292544 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:35.292943 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.293498 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.294244 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:35.294450 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:35.294514 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:35.297436 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:56807: BlockingRecv error: recv got EOF from 127.16.190.195:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:36.037070 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:36.032198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:47057)
0927 21:02:36.032342 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:36.032355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:36.032417 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:36.032576 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:36.032577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:36.032735 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:36.032787 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.032789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.033529 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.033530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.034083 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.034085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.034464 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.034466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.034521 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.035010 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:36.035014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:36.036966 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
W20260927 21:02:36.037130 26325 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:36.032267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:47057 (local address 127.16.190.254:39099)
0927 21:02:36.032388 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:36.032390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:36.032400 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:36.032485 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:36.032486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:36.032496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:36.032568 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:36.032900 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.033432 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.034176 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.034374 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.034429 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.037058 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:47057: BlockingRecv error: recv got EOF from 127.16.190.193:47057 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:36.174638 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:36.163718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:53075)
0927 21:02:36.163907 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:36.163919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:36.163996 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:36.164158 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:36.164158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:36.164351 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:36.164398 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.164400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.165143 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.165145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.165680 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.165682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.166038 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.166039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.166094 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.172630 (+ 6536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:36.172636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:36.174557 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":101,"threads_started":1}
W20260927 21:02:36.174763 26325 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:36.163825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:53075 (local address 127.16.190.254:39099)
0927 21:02:36.163902 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:36.163904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:36.163973 (+ 69us) server_negotiation.cc:407] Connection header received
0927 21:02:36.164054 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:36.164055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:36.164068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:36.164159 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:36.164485 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.165057 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.165779 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.165967 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.166041 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.174671 (+ 8630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:53075: BlockingRecv error: recv got EOF from 127.16.190.194:53075 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:36 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:36.303033 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:36.298068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:48559)
0927 21:02:36.298221 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:36.298239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:36.298301 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:36.298533 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:36.298534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:36.298712 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:36.298761 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.298763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.299510 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.299511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.300044 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:36.300046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.300415 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.300416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:36.300469 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.300974 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:36.300980 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:36.302972 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
W20260927 21:02:36.303179 26325 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:36.298136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:48559 (local address 127.16.190.254:39099)
0927 21:02:36.298225 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:36.298227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:36.298320 (+ 93us) server_negotiation.cc:407] Connection header received
0927 21:02:36.298351 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:36.298352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:36.298364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:36.298436 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:36.298872 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.299383 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.300135 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:36.300324 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:36.300384 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:36.303091 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:48559: BlockingRecv error: recv got EOF from 127.16.190.195:48559 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:37.042783 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:37.037704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:56145)
0927 21:02:37.037852 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:37.037871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:37.037923 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:37.038096 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:37.038097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:37.038264 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:37.038315 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.038316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.039000 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.039001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.039592 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.039594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.040009 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.040010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.040063 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.040572 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:37.040577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:37.042666 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260927 21:02:37.042914 26329 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:37.037790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:56145 (local address 127.16.190.254:39099)
0927 21:02:37.037948 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:37.037950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:37.037960 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:37.037985 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:37.037986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:37.038003 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:37.038075 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:37.038392 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.038915 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.039705 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.039915 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.039982 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.042808 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:56145: BlockingRecv error: recv got EOF from 127.16.190.193:56145 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:37.180398 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:37.175125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:49361)
0927 21:02:37.175300 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:37.175316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:37.175393 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:37.175568 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:37.175569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:37.175760 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:37.175808 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.175810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.176512 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.176513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.177100 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.177103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.177489 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.177490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.177551 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.178014 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:37.178018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:37.180294 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
W20260927 21:02:37.180459 26329 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:37.175187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:49361 (local address 127.16.190.254:39099)
0927 21:02:37.175260 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:37.175262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:37.175378 (+ 116us) server_negotiation.cc:407] Connection header received
0927 21:02:37.175452 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:37.175453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:37.175467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:37.175559 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:37.175891 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.176410 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.177202 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.177399 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.177492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.180394 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:49361: BlockingRecv error: recv got EOF from 127.16.190.194:49361 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:37 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:37.309000 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:37.303625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:59413)
0927 21:02:37.303804 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:37.303815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:37.303897 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:37.304079 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:37.304080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:37.304269 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:37.304315 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.304317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.305154 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.305155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.305690 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:37.305692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.306152 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.306153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:37.306211 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.306747 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:37.306751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:37.308930 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
W20260927 21:02:37.309136 26329 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:37.303814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:59413 (local address 127.16.190.254:39099)
0927 21:02:37.303890 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:37.303892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:37.303903 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:02:37.303970 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:37.303971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:37.303986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:37.304069 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:37.304410 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.305071 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.305791 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:37.306060 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:37.306152 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:37.309053 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:59413: BlockingRecv error: recv got EOF from 127.16.190.195:59413 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:38.048504 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:38.043354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:36205)
0927 21:02:38.043503 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:38.043513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:38.043577 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:38.043748 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:38.043749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:38.043919 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:38.043967 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.043969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.044802 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.044803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.045458 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.045461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.045920 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.045921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.045987 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.046423 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:38.046427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:38.048416 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260927 21:02:38.048651 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:38.043425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:36205 (local address 127.16.190.254:39099)
0927 21:02:38.043572 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:38.043574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:38.043585 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:02:38.043641 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:38.043642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:38.043654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:38.043738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:38.044038 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.044681 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.045557 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.045844 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.045920 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.048527 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:36205: BlockingRecv error: recv got EOF from 127.16.190.193:36205 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:38.186327 26334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:38.181019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:43133)
0927 21:02:38.181212 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:38.181226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:38.181334 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:38.181523 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:38.181524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:38.181715 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:38.181760 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.181762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.182465 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.182466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.183091 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.183093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.183492 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.183493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.183560 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.184067 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:38.184072 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:38.186215 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
W20260927 21:02:38.186468 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:38.181124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43133 (local address 127.16.190.254:39099)
0927 21:02:38.181216 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:38.181219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:38.181313 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:02:38.181423 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:38.181424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:38.181436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:38.181499 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:38.181851 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.182377 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.183190 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.183410 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.183490 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.186383 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:43133: BlockingRecv error: recv got EOF from 127.16.190.194:43133 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:38 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:38.314255 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:38.309506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:44955)
0927 21:02:38.309668 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:38.309681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:38.309749 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:38.309866 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:38.309867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:38.310034 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:38.310080 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.310081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.310733 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.310734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.311265 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:38.311266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.311651 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.311651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:38.311705 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.312166 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:38.312170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:38.314168 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
W20260927 21:02:38.314407 26333 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:38.309551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:44955 (local address 127.16.190.254:39099)
0927 21:02:38.309623 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:38.309626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:38.309750 (+ 124us) server_negotiation.cc:407] Connection header received
0927 21:02:38.309805 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:38.309805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:38.309819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:38.309860 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:38.310149 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.310638 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.311364 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:38.311575 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:38.311645 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:38.314294 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:44955: BlockingRecv error: recv got EOF from 127.16.190.195:44955 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:39.053772 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:39.049128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:36423)
0927 21:02:39.049271 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:39.049282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:39.049330 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:39.049453 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:39.049454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:39.049637 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:39.049685 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.049687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.050368 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.050368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.050920 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.050922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.051307 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.051308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.051354 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.051801 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:39.051805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:39.053713 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260927 21:02:39.053890 26337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:39.049200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:36423 (local address 127.16.190.254:39099)
0927 21:02:39.049350 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:39.049352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:39.049359 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:39.049380 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:39.049380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:39.049390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:39.049449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:39.049782 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.050287 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.051013 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.051232 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.051291 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.053815 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:36423: BlockingRecv error: recv got EOF from 127.16.190.193:36423 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:39.192040 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:39.186856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:36739)
0927 21:02:39.187093 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:39.187105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:39.187203 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:39.187355 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:39.187356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:39.187555 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:39.187600 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.187602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.188275 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.188277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.188875 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.188882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.189252 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.189253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.189309 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.189815 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:39.189823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:39.191921 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
W20260927 21:02:39.192076 26337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:39.186982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:36739 (local address 127.16.190.254:39099)
0927 21:02:39.187085 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:39.187088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:39.187183 (+ 95us) server_negotiation.cc:407] Connection header received
0927 21:02:39.187276 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:39.187277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:39.187289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:39.187340 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:39.187688 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.188191 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.188969 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.189177 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.189254 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.192014 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:36739: BlockingRecv error: recv got EOF from 127.16.190.194:36739 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:39 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:39.319644 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:39.314790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:36141)
0927 21:02:39.314981 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:39.314994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:39.315068 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:39.315199 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:39.315199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:39.315369 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:39.315413 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.315414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.316265 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.316267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.316846 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:39.316848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.317233 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.317234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:39.317287 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.317801 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:39.317806 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:39.319584 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
W20260927 21:02:39.319797 26337 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:39.314847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:36141 (local address 127.16.190.254:39099)
0927 21:02:39.314914 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:39.314916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:39.315060 (+ 144us) server_negotiation.cc:407] Connection header received
0927 21:02:39.315114 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:39.315115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:39.315126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:39.315177 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:39.315490 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.316170 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.316947 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:39.317155 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:39.317215 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:39.319713 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:36141: BlockingRecv error: recv got EOF from 127.16.190.195:36141 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:40.059303 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:40.054408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:49591)
0927 21:02:40.054641 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:40.054653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:40.054745 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:40.054905 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:40.054906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:40.055075 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:40.055122 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.055124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.055760 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.055761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.056330 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.056332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.056740 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.056741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.056784 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.057357 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:40.057362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:40.059226 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
W20260927 21:02:40.059368 26341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:40.054488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:49591 (local address 127.16.190.254:39099)
0927 21:02:40.054625 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:40.054629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:40.054730 (+ 101us) server_negotiation.cc:407] Connection header received
0927 21:02:40.054803 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:40.054804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:40.054818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:40.054875 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:40.055190 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.055686 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.056423 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.056664 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.056723 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.059314 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:49591: BlockingRecv error: recv got EOF from 127.16.190.193:49591 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:40.197957 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:40.192670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:55319)
0927 21:02:40.192870 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:40.192885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:40.192952 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:40.193094 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:40.193094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:40.193274 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:40.193322 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.193323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.194056 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.194057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.194591 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.194592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.194993 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.194995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.195046 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.195803 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:40.195815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:40.197846 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
W20260927 21:02:40.198006 26341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:40.192854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:55319 (local address 127.16.190.254:39099)
0927 21:02:40.192931 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:40.192933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:40.192943 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:02:40.193007 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:40.193008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:40.193019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:40.193073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:40.193388 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.193982 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.194672 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.194877 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.194937 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.197938 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:55319: BlockingRecv error: recv got EOF from 127.16.190.194:55319 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:40 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:40.325152 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:40.320179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:56693)
0927 21:02:40.320318 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:40.320351 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:40.320414 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:40.320596 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:40.320597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:40.320793 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:40.320869 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.320871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.321559 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.321560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.322124 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:40.322126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.322512 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.322513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:40.322577 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.323159 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:40.323164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:40.325072 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260927 21:02:40.325345 26341 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:40.320304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:56693 (local address 127.16.190.254:39099)
0927 21:02:40.320379 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:40.320382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:40.320394 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:40.320504 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:40.320505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:40.320520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:40.320587 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:40.320954 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.321460 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.322214 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:40.322407 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:40.322513 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:40.325250 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:56693: BlockingRecv error: recv got EOF from 127.16.190.195:56693 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:41.065201 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:41.059999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:50703)
0927 21:02:41.060160 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:41.060178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:41.060239 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:41.060416 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:41.060417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:41.060598 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:41.060658 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.060659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.061406 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.061407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.061955 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.061957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.062340 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.062341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.062448 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.063010 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:41.063015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:41.065140 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
W20260927 21:02:41.065369 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:41.060097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:50703 (local address 127.16.190.254:39099)
0927 21:02:41.060264 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:41.060266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:41.060278 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:02:41.060312 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:41.060313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:41.060323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:41.060397 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:41.060768 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.061314 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.062055 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.062265 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.062323 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.065283 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:50703: BlockingRecv error: recv got EOF from 127.16.190.193:50703 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:41.211038 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:41.198576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:51563)
0927 21:02:41.198799 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:41.198811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:41.198885 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:41.199038 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:41.199039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:41.199213 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:41.199259 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.199261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.200012 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.200013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.200544 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.200545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.201024 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.201025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.201070 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.208552 (+ 7482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:41.208559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:41.210975 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
W20260927 21:02:41.211252 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:41.198729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51563 (local address 127.16.190.254:39099)
0927 21:02:41.198800 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:41.198802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:41.198868 (+ 66us) server_negotiation.cc:407] Connection header received
0927 21:02:41.198963 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:41.198964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:41.198978 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:41.199022 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:41.199366 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.199923 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.200653 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.200951 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.201023 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.211175 (+ 10152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:51563: BlockingRecv error: recv got EOF from 127.16.190.194:51563 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:41 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.195@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:41.330816 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:41.325809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.195:33795)
0927 21:02:41.325993 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:41.326007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:41.326081 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:41.326218 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:41.326219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:41.326393 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:41.326445 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.326447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.327095 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.327097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.327909 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:41.327912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.328369 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.328370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:41.328418 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.328982 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:41.328987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:41.330755 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
W20260927 21:02:41.330984 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:41.325878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:33795 (local address 127.16.190.254:39099)
0927 21:02:41.325991 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:41.325993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:41.326085 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:02:41.326139 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:41.326140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:41.326152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:41.326195 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:41.326511 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.327012 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.328008 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:41.328283 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:41.328336 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:41.330882 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.195:33795: BlockingRecv error: recv got EOF from 127.16.190.195:33795 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 21:02:42 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:42 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.193@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:42.070541 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.065871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.193:43667)
0927 21:02:42.066051 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:42.066063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:42.066115 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:42.066308 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:42.066310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:42.066481 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:42.066543 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.066545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.067229 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.067230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.067779 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.067780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.068150 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.068151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.068194 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.068636 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:42.068640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:42.070470 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
W20260927 21:02:42.070712 26349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:42.065991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:43667 (local address 127.16.190.254:39099)
0927 21:02:42.066147 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.066149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.066158 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:42.066182 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.066183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.066192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.066263 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.066622 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.067119 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.067880 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.068077 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.068135 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.070602 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.193:43667: BlockingRecv error: recv got EOF from 127.16.190.193:43667 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
Sep 27 21:02:42 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 21:02:42 dist-test-slave-6c7s krb5kdc[25290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.190.194@KRBTEST.COM for oryx/127.16.190.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 21:02:42.217275 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.211785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39099 (local address 127.16.190.194:46191)
0927 21:02:42.211994 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:42.212006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:42.212095 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:42.212267 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:42.212267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:42.212442 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:42.212527 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.212529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.213246 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.213247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.213840 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.213843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.214275 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.214277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.214329 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.215049 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:42.215054 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:42.217165 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.190.254:39099: Server oryx/127.16.190.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":110,"threads_started":1}
W20260927 21:02:42.217327 26349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:42.211946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:46191 (local address 127.16.190.254:39099)
0927 21:02:42.212011 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.212013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.212102 (+ 89us) server_negotiation.cc:407] Connection header received
0927 21:02:42.212167 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.212167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.212180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.212244 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.212608 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.213128 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.213950 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.214166 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.214239 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.217257 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.190.194:46191: BlockingRecv error: recv got EOF from 127.16.190.194:46191 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:02:42.237665 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25873
I20260927 21:02:42.242573 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26027
I20260927 21:02:42.246964 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26159
I20260927 21:02:42.251401 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 25803
2026-09-27T21:02:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17500 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T21:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:42Z Disabled control of system clock
I20260927 21:02:42.279718 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40629
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:40629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 21:02:42.360926 26354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:42.361140 26354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:42.361161 26354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:42.362279 26354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:42.362316 26354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:42.362329 26354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:42.362341 26354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:42.362352 26354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:42.363922 26354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:40629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:42.364147 26354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:42.364394 26354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:42.369385 26360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.369421 26359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.369432 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.369481 26354 server_base.cc:1034] running on GCE node
I20260927 21:02:42.369880 26354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:42.370229 26354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:42.371423 26354 hybrid_clock.cc:648] HybridClock initialized: now 1790542962371405 us; error 44 us; skew 500 ppm
I20260927 21:02:42.373013 26354 webserver.cc:460] Webserver started at http://127.16.190.254:43473/ using document root <none> and password file <none>
I20260927 21:02:42.373286 26354 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:42.373358 26354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:42.373574 26354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:42.374763 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "7c8f30a2fae840919b6f9d0c2f829a2f"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "55fd98f2d4416229407d067aa8493040"
server_key_iv: "3e73cb8f49ea978afd7b80e0d4a277eb"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.375151 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "7c8f30a2fae840919b6f9d0c2f829a2f"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "55fd98f2d4416229407d067aa8493040"
server_key_iv: "3e73cb8f49ea978afd7b80e0d4a277eb"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.376715 26354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:42.377666 26368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.377892 26354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:42.378003 26354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "7c8f30a2fae840919b6f9d0c2f829a2f"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "55fd98f2d4416229407d067aa8493040"
server_key_iv: "3e73cb8f49ea978afd7b80e0d4a277eb"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.378098 26354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:42.389985 26354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:42.390352 26354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:42.390515 26354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:42.394796 26354 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:40629
I20260927 21:02:42.394831 26420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:40629 every 8 connection(s)
I20260927 21:02:42.395192 26354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:42.395922 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:42.398664 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f: Bootstrap starting.
I20260927 21:02:42.399261 26421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:42.399567 26421 log.cc:824] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:42.400331 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f: No bootstrap required, opened a new log
I20260927 21:02:42.402091 26421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } }
I20260927 21:02:42.402240 26421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:42.402263 26421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8f30a2fae840919b6f9d0c2f829a2f, State: Initialized, Role: FOLLOWER
I20260927 21:02:42.402426 26421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } }
I20260927 21:02:42.402506 26421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:42.402529 26421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:42.402556 26421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:42.403144 26421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } }
I20260927 21:02:42.403235 26421 leader_election.cc:302] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c8f30a2fae840919b6f9d0c2f829a2f; no voters:
I20260927 21:02:42.403450 26421 leader_election.cc:288] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:42.403607 26424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:42.403728 26421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:42.403776 26424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [term 1 LEADER]: Becoming Leader. State: Replica: 7c8f30a2fae840919b6f9d0c2f829a2f, State: Running, Role: LEADER
I20260927 21:02:42.403875 26424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } }
I20260927 21:02:42.404276 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26354
I20260927 21:02:42.404376 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:42.404593 17147 external_mini_cluster.cc:1428] Setting key 7fd7b2d8fe6b48036a572c5082631a6a
I20260927 21:02:42.404596 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c8f30a2fae840919b6f9d0c2f829a2f. Latest consensus state: current_term: 1 leader_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } } }
I20260927 21:02:42.404670 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:42.405022 26433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:42.405171 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c8f30a2fae840919b6f9d0c2f829a2f" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40629 } } }
I20260927 21:02:42.405584 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:42.405628 26433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:42.407451 26433 catalog_manager.cc:1347] Generated new cluster ID: 844a806c39f54e0a9b07794d1656694a
I20260927 21:02:42.407519 26433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:42.411597 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.405386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.16.190.254:40629)
0927 21:02:42.405933 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.405941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.405971 (+ 30us) server_negotiation.cc:407] Connection header received
0927 21:02:42.406302 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.406314 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.406399 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.406561 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.406833 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.408208 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.409310 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.410013 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.410146 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.410865 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:42.410869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:42.410869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:42.411089 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:42.411134 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:42.411149 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:42.411221 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:42.411288 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:02:42.411372 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":486,"thread_start_us":65,"threads_started":1}
I20260927 21:02:42.429550 26433 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:42.430125 26433 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:42.434266 26433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c8f30a2fae840919b6f9d0c2f829a2f: Generated new TSK 0
I20260927 21:02:42.434523 26433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:42.441439 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--builtin_ntp_servers=127.16.190.212:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 21:02:42.531414 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:42.531669 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:42.531706 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:42.532919 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:42.532989 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:42.533031 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:42.534442 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:42.534746 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:42.535077 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:42.540555 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.540632 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.540736 26445 server_base.cc:1034] running on GCE node
W20260927 21:02:42.540858 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.541105 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:42.541371 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:42.542531 26445 hybrid_clock.cc:648] HybridClock initialized: now 1790542962542518 us; error 45 us; skew 500 ppm
I20260927 21:02:42.544067 26445 webserver.cc:460] Webserver started at http://127.16.190.193:35761/ using document root <none> and password file <none>
I20260927 21:02:42.544378 26445 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:42.544461 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:42.544591 26445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:42.546082 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "d58a094cdf5e42aaade091758ed6b500"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "c7d9a4ccb1fc6b84d665d97b6f476592"
server_key_iv: "e6b4faa819621e3b9bbe72850c1257d5"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.546482 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "d58a094cdf5e42aaade091758ed6b500"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "c7d9a4ccb1fc6b84d665d97b6f476592"
server_key_iv: "e6b4faa819621e3b9bbe72850c1257d5"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.548102 26445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:42.549098 26459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.549259 26445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:42.549377 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "d58a094cdf5e42aaade091758ed6b500"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "c7d9a4ccb1fc6b84d665d97b6f476592"
server_key_iv: "e6b4faa819621e3b9bbe72850c1257d5"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.549474 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:42.567924 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:42.568300 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:42.568464 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:42.568722 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:42.569125 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:42.569182 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.569217 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:42.569244 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.575541 26445 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:35751
I20260927 21:02:42.575591 26572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:35751 every 8 connection(s)
I20260927 21:02:42.576009 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:42.577366 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26445
I20260927 21:02:42.577526 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:42.577744 17147 external_mini_cluster.cc:1428] Setting key edf38ee69bd641aefc4ff351456d4fb8
I20260927 21:02:42.578697 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--builtin_ntp_servers=127.16.190.212:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:02:42.582021 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.576846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:34807 (local address 127.16.190.254:40629)
0927 21:02:42.576905 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.576907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.577144 (+ 237us) server_negotiation.cc:407] Connection header received
0927 21:02:42.577824 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.577825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.577842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.577904 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.578739 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.579302 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.580335 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.580565 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.580648 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.581654 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:42.581655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:42.581656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:42.581661 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:42.581681 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:42.581686 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:42.581751 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:42.581818 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:02:42.581850 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:02:42.582211 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.576777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40629 (local address 127.16.190.193:34807)
0927 21:02:42.576987 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:42.577045 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:42.577713 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:42.577981 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:42.577985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:42.577998 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:42.578650 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.578656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.579405 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.579406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.580221 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.580223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.580654 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.580656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.580857 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.581482 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:42.581482 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:42.581514 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:42.581537 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:42.581757 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:42.581761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:42.581764 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:42.581819 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 21:02:42.581901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260927 21:02:42.582738 26573 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40629
I20260927 21:02:42.582834 26573 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:42.583045 26573 heartbeater.cc:511] Master 127.16.190.254:40629 requested a full tablet report, sending...
I20260927 21:02:42.583581 26385 ts_manager.cc:194] Registered new tserver with Master: d58a094cdf5e42aaade091758ed6b500 (127.16.190.193:35751)
I20260927 21:02:42.584292 26385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.193:34807
W20260927 21:02:42.662784 26576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:42.663076 26576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:42.663115 26576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:42.664448 26576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:42.664556 26576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:42.664598 26576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:42.666128 26576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:42.666478 26576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:42.666826 26576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:42.672421 26581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.672482 26582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.672695 26576 server_base.cc:1034] running on GCE node
W20260927 21:02:42.672429 26584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.673008 26576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:42.673262 26576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:42.674404 26576 hybrid_clock.cc:648] HybridClock initialized: now 1790542962674385 us; error 42 us; skew 500 ppm
I20260927 21:02:42.675637 26576 webserver.cc:460] Webserver started at http://127.16.190.194:45857/ using document root <none> and password file <none>
I20260927 21:02:42.675848 26576 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:42.675890 26576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:42.675982 26576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:42.677114 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "afe5c7830b10440ea99b1959b6d999ae"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "8b0cf1f0e84b52dcfb5833f770e93758"
server_key_iv: "8f5ddc2988fbd4b0616b593327237d06"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.677423 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "afe5c7830b10440ea99b1959b6d999ae"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "8b0cf1f0e84b52dcfb5833f770e93758"
server_key_iv: "8f5ddc2988fbd4b0616b593327237d06"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.678674 26576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:02:42.679338 26590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.679544 26576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:02:42.679631 26576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "afe5c7830b10440ea99b1959b6d999ae"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "8b0cf1f0e84b52dcfb5833f770e93758"
server_key_iv: "8f5ddc2988fbd4b0616b593327237d06"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.679697 26576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:42.692066 26576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:42.692348 26576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:42.692452 26576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:42.692624 26576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:42.693028 26576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:42.693068 26576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.693090 26576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:42.693104 26576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.699015 26576 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:39591
I20260927 21:02:42.699090 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:39591 every 8 connection(s)
I20260927 21:02:42.699450 26576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:42.704128 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26576
I20260927 21:02:42.704241 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:42.704492 17147 external_mini_cluster.cc:1428] Setting key a126dbdac26178f6d17219dd5ac31d72
I20260927 21:02:42.704983 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.700271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:37623 (local address 127.16.190.254:40629)
0927 21:02:42.700360 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.700362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.700563 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:02:42.701148 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.701150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.701162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.701223 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.701893 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.702425 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.703389 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.703608 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.703673 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.704675 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:42.704676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:42.704677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:42.704682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:42.704697 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:42.704702 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:42.704775 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:42.704873 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 21:02:42.704906 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:02:42.705222 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.700225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40629 (local address 127.16.190.194:37623)
0927 21:02:42.700435 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:42.700472 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:42.701033 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:42.701302 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:42.701306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:42.701314 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:42.701794 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.701798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.702507 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.702508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.703292 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.703294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.703693 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.703694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.703820 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.704521 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:42.704521 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:42.704550 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:42.704590 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:42.704775 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:42.704782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:42.704785 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:42.704878 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 21:02:42.704981 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260927 21:02:42.705560 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--builtin_ntp_servers=127.16.190.212:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 21:02:42.706723 26704 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40629
I20260927 21:02:42.706845 26704 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:42.707037 26704 heartbeater.cc:511] Master 127.16.190.254:40629 requested a full tablet report, sending...
I20260927 21:02:42.707509 26385 ts_manager.cc:194] Registered new tserver with Master: afe5c7830b10440ea99b1959b6d999ae (127.16.190.194:39591)
I20260927 21:02:42.707994 26385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.194:37623
W20260927 21:02:42.789232 26707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:42.789484 26707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:42.789531 26707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:42.790759 26707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:42.790827 26707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:42.790868 26707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:42.792259 26707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:40629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:42.792546 26707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:42.792883 26707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:42.798209 26715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.798321 26712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.798321 26713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.798887 26707 server_base.cc:1034] running on GCE node
I20260927 21:02:42.799157 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:42.799401 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:42.800545 26707 hybrid_clock.cc:648] HybridClock initialized: now 1790542962800522 us; error 43 us; skew 500 ppm
I20260927 21:02:42.802053 26707 webserver.cc:460] Webserver started at http://127.16.190.195:46175/ using document root <none> and password file <none>
I20260927 21:02:42.802302 26707 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:42.802349 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:42.802448 26707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:42.803711 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "629d2a13c5814b8e86abef6a1910f759"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "195d7682d07f26c5435360d78cd445aa"
server_key_iv: "9b74a1146e17216e044e6d3f4e8db4c8"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.804107 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "629d2a13c5814b8e86abef6a1910f759"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "195d7682d07f26c5435360d78cd445aa"
server_key_iv: "9b74a1146e17216e044e6d3f4e8db4c8"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.806182 26707 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 21:02:42.807120 26721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.807435 26707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:42.807622 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "629d2a13c5814b8e86abef6a1910f759"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "195d7682d07f26c5435360d78cd445aa"
server_key_iv: "9b74a1146e17216e044e6d3f4e8db4c8"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.807729 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:42.832377 26707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:42.832763 26707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:42.832978 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:42.833184 26707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:42.833475 26707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:42.833508 26707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.833530 26707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:42.833542 26707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.839263 26707 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:44239
I20260927 21:02:42.839340 26834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:44239 every 8 connection(s)
I20260927 21:02:42.839694 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:42.841318 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26707
I20260927 21:02:42.841415 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:42.841635 17147 external_mini_cluster.cc:1428] Setting key 33775ca8fa550cef69794afda6fe6f80
I20260927 21:02:42.845613 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.840501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:34657 (local address 127.16.190.254:40629)
0927 21:02:42.840568 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.840570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.840797 (+ 227us) server_negotiation.cc:407] Connection header received
0927 21:02:42.841575 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.841576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.841592 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.841654 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.842619 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.843121 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.844183 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.844371 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.844432 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.845388 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:42.845389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:42.845389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:42.845394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:42.845408 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:42.845412 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:42.845462 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:42.845525 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 21:02:42.845552 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:02:42.845820 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:42.840464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:40629 (local address 127.16.190.195:34657)
0927 21:02:42.840667 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:42.840702 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:42.841413 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:42.841739 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:42.841743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:42.841754 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:42.842503 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.842511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.843216 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.843217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.844086 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:42.844089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:42.844452 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:42.844453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:42.844566 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:42.845194 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:42.845194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:42.845222 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:42.845245 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:42.845472 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:42.845475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:42.845478 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:42.845524 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:02:42.845602 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260927 21:02:42.846297 26835 heartbeater.cc:347] Connected to a master server at 127.16.190.254:40629
I20260927 21:02:42.846401 26835 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:42.846580 26835 heartbeater.cc:511] Master 127.16.190.254:40629 requested a full tablet report, sending...
I20260927 21:02:42.847007 26385 ts_manager.cc:194] Registered new tserver with Master: 629d2a13c5814b8e86abef6a1910f759 (127.16.190.195:44239)
I20260927 21:02:42.847381 26385 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.195:34657
I20260927 21:02:42.853579 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:02:42.855584 26443 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:42.855065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.16.190.254:40629)
0927 21:02:42.855123 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:42.855124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:42.855192 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:02:42.855254 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:42.855255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:42.855266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:42.855318 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:42.855445 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54412: BlockingRecv error: recv got EOF from 127.0.0.1:54412 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:02:42.855643 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26445
I20260927 21:02:42.860164 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26576
I20260927 21:02:42.864727 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26707
I20260927 21:02:42.869122 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26354
2026-09-27T21:02:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (616 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T21:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:42Z Disabled control of system clock
I20260927 21:02:42.896378 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:33349
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:02:42.980718 26846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:42.981032 26846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:42.981056 26846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:42.982188 26846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:42.982226 26846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:42.982239 26846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:42.982254 26846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:42.982265 26846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:42.983562 26846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:33349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:42.983768 26846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:42.984018 26846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:42.989221 26854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.989216 26851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:42.989219 26852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:42.989447 26846 server_base.cc:1034] running on GCE node
I20260927 21:02:42.989768 26846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:42.990028 26846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:42.991168 26846 hybrid_clock.cc:648] HybridClock initialized: now 1790542962991148 us; error 45 us; skew 500 ppm
I20260927 21:02:42.992403 26846 webserver.cc:460] Webserver started at http://127.16.190.254:33545/ using document root <none> and password file <none>
I20260927 21:02:42.992702 26846 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:42.992751 26846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:42.992910 26846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:42.994048 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "f208093c3c1540338f9834605f8762db"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "e5d4308d0ce80581358b7ff58b0890b2"
server_key_iv: "8186720d3beea253b6c3cf97535492e9"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.994411 26846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "f208093c3c1540338f9834605f8762db"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "e5d4308d0ce80581358b7ff58b0890b2"
server_key_iv: "8186720d3beea253b6c3cf97535492e9"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.995775 26846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:42.996574 26860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:42.996755 26846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:42.996871 26846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "f208093c3c1540338f9834605f8762db"
format_stamp: "Formatted at 2026-09-27 21:02:42 on dist-test-slave-6c7s"
server_key: "e5d4308d0ce80581358b7ff58b0890b2"
server_key_iv: "8186720d3beea253b6c3cf97535492e9"
server_key_version: "encryptionkey@0"
I20260927 21:02:42.996963 26846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:43.000396 26846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:43.000672 26846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:43.000869 26846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:43.004733 26846 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:33349
I20260927 21:02:43.004766 26912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:33349 every 8 connection(s)
I20260927 21:02:43.005141 26846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:43.005777 26913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:43.008641 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db: Bootstrap starting.
I20260927 21:02:43.009310 26913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:43.009616 26913 log.cc:824] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:43.010366 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db: No bootstrap required, opened a new log
I20260927 21:02:43.011305 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26846
I20260927 21:02:43.011399 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:43.011602 17147 external_mini_cluster.cc:1428] Setting key cffe1aa726c22fab1fa155dfa122ba98
I20260927 21:02:43.012571 26913 raft_consensus.cc:348] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } }
I20260927 21:02:43.012763 26913 raft_consensus.cc:374] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:43.012844 26913 raft_consensus.cc:729] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f208093c3c1540338f9834605f8762db, State: Initialized, Role: FOLLOWER
I20260927 21:02:43.013384 26913 consensus_queue.cc:260] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } }
I20260927 21:02:43.013506 26913 raft_consensus.cc:388] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:43.013533 26913 raft_consensus.cc:482] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:43.013607 26913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:43.014221 26913 raft_consensus.cc:504] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } }
I20260927 21:02:43.014325 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.012207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.16.190.254:33349)
0927 21:02:43.012418 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:43.012426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:43.012450 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:02:43.012547 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:43.012552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:43.012592 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:43.012704 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:43.013437 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:43.013440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:43.013441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:43.013444 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:43.013478 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:43.013488 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:43.013578 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:43.013780 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 21:02:43.013859 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260927 21:02:43.014312 26913 leader_election.cc:302] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f208093c3c1540338f9834605f8762db; no voters:
I20260927 21:02:43.014488 26913 leader_election.cc:288] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:43.014528 26918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:43.014742 26913 sys_catalog.cc:564] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:43.014936 26918 raft_consensus.cc:686] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [term 1 LEADER]: Becoming Leader. State: Replica: f208093c3c1540338f9834605f8762db, State: Running, Role: LEADER
I20260927 21:02:43.015121 26918 consensus_queue.cc:237] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } }
I20260927 21:02:43.015491 26918 sys_catalog.cc:455] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f208093c3c1540338f9834605f8762db. Latest consensus state: current_term: 1 leader_uuid: "f208093c3c1540338f9834605f8762db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } } }
I20260927 21:02:43.015504 26920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f208093c3c1540338f9834605f8762db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f208093c3c1540338f9834605f8762db" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 33349 } } }
I20260927 21:02:43.015661 26918 sys_catalog.cc:458] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:43.015674 26920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:43.016168 26929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:43.016780 26929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:43.018335 26929 catalog_manager.cc:1347] Generated new cluster ID: d761ac38e1134e04be903c26fd9cf38c
I20260927 21:02:43.018390 26929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:02:43.031637 26929 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:43.031832 26929 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:43.043403 26929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f208093c3c1540338f9834605f8762db: Generated new TSK 0
I20260927 21:02:43.043713 26929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:02:43.054361 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--builtin_ntp_servers=127.16.190.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:02:43.144059 26937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:43.144295 26937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:43.144321 26937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:43.145828 26937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:43.145885 26937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:43.145915 26937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:43.147382 26937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:43.147676 26937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:43.147995 26937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:43.154778 26943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:43.154896 26937 server_base.cc:1034] running on GCE node
W20260927 21:02:43.154722 26942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:43.154722 26945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:43.155216 26937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:43.155493 26937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:43.156667 26937 hybrid_clock.cc:648] HybridClock initialized: now 1790542963156657 us; error 53 us; skew 500 ppm
I20260927 21:02:43.158350 26937 webserver.cc:460] Webserver started at http://127.16.190.193:44011/ using document root <none> and password file <none>
I20260927 21:02:43.158591 26937 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:43.158636 26937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:43.158731 26937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:43.159811 26937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "2f4875f403794a8b86ec20de92bc5079"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "77779033000054aa19ac0d8cf8c5b965"
server_key_iv: "c5f33344ed057463bda46c235d52e0be"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.160149 26937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "2f4875f403794a8b86ec20de92bc5079"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "77779033000054aa19ac0d8cf8c5b965"
server_key_iv: "c5f33344ed057463bda46c235d52e0be"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.161618 26937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:02:43.162573 26951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.162845 26937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:43.162984 26937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "2f4875f403794a8b86ec20de92bc5079"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "77779033000054aa19ac0d8cf8c5b965"
server_key_iv: "c5f33344ed057463bda46c235d52e0be"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.163095 26937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:43.166785 26937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:43.167086 26937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:43.167233 26937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:43.167450 26937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:43.167774 26937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:43.167829 26937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.167871 26937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:43.167898 26937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.174404 26937 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:41505
I20260927 21:02:43.174444 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:41505 every 8 connection(s)
I20260927 21:02:43.174806 26937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:43.178041 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.175533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:47383 (local address 127.16.190.254:33349)
0927 21:02:43.175613 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:43.175615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:43.175854 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:02:43.176410 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:43.176411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:43.176417 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:43.176484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:43.177797 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:43.177798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:43.177798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:43.177802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:43.177820 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:43.177824 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:43.177880 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:43.177967 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 21:02:43.177981 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:02:43.178315 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.175529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:33349 (local address 127.16.190.193:47383)
0927 21:02:43.175729 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:43.175764 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:43.176280 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:43.176565 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:43.176568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:43.176577 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:43.177631 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:43.177631 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:43.177665 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:43.177699 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:43.177892 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:43.177897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:43.177900 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:43.177964 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:02:43.178018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260927 21:02:43.178845 27065 heartbeater.cc:347] Connected to a master server at 127.16.190.254:33349
I20260927 21:02:43.178960 27065 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:43.179160 27065 heartbeater.cc:511] Master 127.16.190.254:33349 requested a full tablet report, sending...
I20260927 21:02:43.179675 26877 ts_manager.cc:194] Registered new tserver with Master: 2f4875f403794a8b86ec20de92bc5079 (127.16.190.193:41505)
I20260927 21:02:43.180128 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 26937
I20260927 21:02:43.180231 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:43.180433 17147 external_mini_cluster.cc:1428] Setting key 5d5dba192a2a7e80338627a6d2ef934f
I20260927 21:02:43.181458 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--builtin_ntp_servers=127.16.190.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:02:43.263998 27068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:43.264271 27068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:43.264312 27068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:43.265609 27068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:43.265681 27068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:43.265728 27068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:43.267267 27068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:43.267580 27068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:43.267891 27068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:43.273135 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:43.273154 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:43.273137 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:43.273379 27068 server_base.cc:1034] running on GCE node
I20260927 21:02:43.273599 27068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:43.273856 27068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:43.275053 27068 hybrid_clock.cc:648] HybridClock initialized: now 1790542963275049 us; error 45 us; skew 500 ppm
I20260927 21:02:43.276501 27068 webserver.cc:460] Webserver started at http://127.16.190.194:34791/ using document root <none> and password file <none>
I20260927 21:02:43.276773 27068 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:43.276891 27068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:43.277052 27068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:43.278184 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "94c5786b11c34616bb93d29edc57f91c"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "1b218396c22afed144c9a7f269bcbbd6"
server_key_iv: "204a1aeeea7ac32a95e5fa6f28d848e5"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.278553 27068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "94c5786b11c34616bb93d29edc57f91c"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "1b218396c22afed144c9a7f269bcbbd6"
server_key_iv: "204a1aeeea7ac32a95e5fa6f28d848e5"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.279959 27068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:43.280885 27082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.281144 27068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:02:43.281252 27068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "94c5786b11c34616bb93d29edc57f91c"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "1b218396c22afed144c9a7f269bcbbd6"
server_key_iv: "204a1aeeea7ac32a95e5fa6f28d848e5"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.281345 27068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:43.284896 27068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:43.285202 27068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:43.285351 27068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:43.285565 27068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:43.285899 27068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:43.285962 27068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.286006 27068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:43.286034 27068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.292053 27068 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:36419
I20260927 21:02:43.292097 27195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:36419 every 8 connection(s)
I20260927 21:02:43.292482 27068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:02:43.295806 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.293340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51679 (local address 127.16.190.254:33349)
0927 21:02:43.293419 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:43.293422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:43.293737 (+ 315us) server_negotiation.cc:407] Connection header received
0927 21:02:43.294192 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:43.294194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:43.294199 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:43.294271 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:43.295558 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:43.295560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:43.295560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:43.295565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:43.295584 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:43.295589 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:43.295652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:43.295726 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 21:02:43.295741 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:02:43.295919 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.293316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:33349 (local address 127.16.190.194:51679)
0927 21:02:43.293528 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:43.293616 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:43.294088 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:43.294346 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:43.294349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:43.294357 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:43.295390 (+ 1033us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:43.295391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:43.295421 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:43.295452 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:43.295656 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:43.295658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:43.295661 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:43.295713 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 21:02:43.295762 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260927 21:02:43.296229 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27068
I20260927 21:02:43.296332 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:43.296489 27196 heartbeater.cc:347] Connected to a master server at 127.16.190.254:33349
I20260927 21:02:43.296555 17147 external_mini_cluster.cc:1428] Setting key 310ba9bce800d4fb6ee38dd8439691fc
I20260927 21:02:43.296595 27196 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:43.296751 27196 heartbeater.cc:511] Master 127.16.190.254:33349 requested a full tablet report, sending...
I20260927 21:02:43.297169 26876 ts_manager.cc:194] Registered new tserver with Master: 94c5786b11c34616bb93d29edc57f91c (127.16.190.194:36419)
I20260927 21:02:43.297788 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--builtin_ntp_servers=127.16.190.212:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 21:02:43.381953 27199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:43.382263 27199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:43.382292 27199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:43.383569 27199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:43.383616 27199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:43.383646 27199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:43.385285 27199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:43.385578 27199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:43.385857 27199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:43.391768 27204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:43.391757 27205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:43.391793 27199 server_base.cc:1034] running on GCE node
W20260927 21:02:43.391760 27207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:43.392247 27199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:43.392544 27199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:43.393718 27199 hybrid_clock.cc:648] HybridClock initialized: now 1790542963393706 us; error 32 us; skew 500 ppm
I20260927 21:02:43.395051 27199 webserver.cc:460] Webserver started at http://127.16.190.195:38541/ using document root <none> and password file <none>
I20260927 21:02:43.395327 27199 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:43.395373 27199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:43.395548 27199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:43.396690 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "bfb9a350ef27417cbd766cd9d9a41387"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "9b9368672f92a05abba301d69d61e5fa"
server_key_iv: "fe7a5eed6c059aa1e30f049af9cf2e81"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.397128 27199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "bfb9a350ef27417cbd766cd9d9a41387"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "9b9368672f92a05abba301d69d61e5fa"
server_key_iv: "fe7a5eed6c059aa1e30f049af9cf2e81"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.398447 27199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:43.399212 27213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.399425 27199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:43.399515 27199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "bfb9a350ef27417cbd766cd9d9a41387"
format_stamp: "Formatted at 2026-09-27 21:02:43 on dist-test-slave-6c7s"
server_key: "9b9368672f92a05abba301d69d61e5fa"
server_key_iv: "fe7a5eed6c059aa1e30f049af9cf2e81"
server_key_version: "encryptionkey@0"
I20260927 21:02:43.399610 27199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:43.403285 27199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:43.403570 27199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:43.403710 27199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:43.403935 27199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:43.404289 27199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:43.404345 27199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.404385 27199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:43.404412 27199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:43.410478 27199 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:36327
I20260927 21:02:43.410538 27326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:36327 every 8 connection(s)
I20260927 21:02:43.410928 27199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:43.412770 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27199
I20260927 21:02:43.412899 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:43.413115 17147 external_mini_cluster.cc:1428] Setting key b1b9424d05b88a7091892bfcb74bcfd0
I20260927 21:02:43.414472 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.411788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:44011 (local address 127.16.190.254:33349)
0927 21:02:43.411867 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:43.411869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:43.412123 (+ 254us) server_negotiation.cc:407] Connection header received
0927 21:02:43.412635 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:43.412637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:43.412647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:43.412745 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:43.414285 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:43.414286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:43.414286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:02:43.414291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:43.414306 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:02:43.414310 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:43.414351 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:43.414404 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:02:43.414418 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:02:43.414711 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:43.411706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:33349 (local address 127.16.190.195:44011)
0927 21:02:43.411966 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:43.412011 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:43.412512 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:43.412803 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:43.412847 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:43.412858 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:43.414109 (+ 1251us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:02:43.414110 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:43.414154 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:02:43.414194 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:43.414352 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:43.414354 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:43.414357 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:43.414411 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:02:43.414463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260927 21:02:43.415232 27327 heartbeater.cc:347] Connected to a master server at 127.16.190.254:33349
I20260927 21:02:43.415338 27327 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:43.415462 27327 heartbeater.cc:511] Master 127.16.190.254:33349 requested a full tablet report, sending...
I20260927 21:02:43.415817 26876 ts_manager.cc:194] Registered new tserver with Master: bfb9a350ef27417cbd766cd9d9a41387 (127.16.190.195:36327)
I20260927 21:02:43.425355 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 21:02:43.427368 26917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:02:43.426855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.16.190.254:33349)
0927 21:02:43.426921 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:43.426923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:43.427027 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:02:43.427073 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:43.427073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:43.427078 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:43.427140 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:43.427272 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43194: BlockingRecv error: recv got EOF from 127.0.0.1:43194 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:02:43.427431 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26937
I20260927 21:02:43.432276 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27068
I20260927 21:02:43.436599 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27199
I20260927 21:02:43.440971 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 26846
2026-09-27T21:02:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:43 dist-test-slave-6c7s krb5kdc[27337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:43 dist-test-slave-6c7s krb5kdc[27337](info): set up 2 sockets
Sep 27 21:02:43 dist-test-slave-6c7s krb5kdc[27337](info): commencing operation
krb5kdc: starting...
W20260927 21:02:44.498534 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:44 dist-test-slave-6c7s krb5kdc[27337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:02:44.628377 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45801
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:44.710716 27350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:44.710973 27350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:44.711011 27350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:44.712195 27350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:02:44.712271 27350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:44.712298 27350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:44.712329 27350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:02:44.712352 27350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:02:44.713970 27350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:44.714274 27350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:44.714514 27350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:44.719878 27356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:44.719878 27355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:44.720013 27350 server_base.cc:1034] running on GCE node
W20260927 21:02:44.720098 27358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:44.720368 27350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:44.720685 27350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:44.721877 27350 hybrid_clock.cc:648] HybridClock initialized: now 1790542964721862 us; error 47 us; skew 500 ppm
Sep 27 21:02:44 dist-test-slave-6c7s krb5kdc[27337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542964, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:44.724597 27350 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:02:44.725147 27350 webserver.cc:460] Webserver started at http://127.16.190.254:39445/ using document root <none> and password file <none>
I20260927 21:02:44.725410 27350 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:44.725481 27350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:44.725631 27350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:44.726730 27350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "acfb174d48f14c2eb11cc70484296dd7"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "8ff2b1323d51c9c4de7afc4175e22d1d"
server_key_iv: "e0e4bdb61f359496959242ff9e542db0"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.727093 27350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "acfb174d48f14c2eb11cc70484296dd7"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "8ff2b1323d51c9c4de7afc4175e22d1d"
server_key_iv: "e0e4bdb61f359496959242ff9e542db0"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.728448 27350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:44.729415 27365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:44.729779 27350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:02:44.729895 27350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "acfb174d48f14c2eb11cc70484296dd7"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "8ff2b1323d51c9c4de7afc4175e22d1d"
server_key_iv: "e0e4bdb61f359496959242ff9e542db0"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.729978 27350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:44.736681 27350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:44.738931 27350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:44.739113 27350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:44.743252 27350 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:45801
I20260927 21:02:44.743299 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:45801 every 8 connection(s)
I20260927 21:02:44.743695 27350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:02:44.744872 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:44.747215 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: Bootstrap starting.
I20260927 21:02:44.747823 27418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:44.748121 27418 log.cc:824] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:44.748955 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: No bootstrap required, opened a new log
I20260927 21:02:44.750569 27418 raft_consensus.cc:348] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } }
I20260927 21:02:44.750727 27418 raft_consensus.cc:374] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:44.750751 27418 raft_consensus.cc:729] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfb174d48f14c2eb11cc70484296dd7, State: Initialized, Role: FOLLOWER
I20260927 21:02:44.750907 27418 consensus_queue.cc:260] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } }
I20260927 21:02:44.750984 27418 raft_consensus.cc:388] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:02:44.751008 27418 raft_consensus.cc:482] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:02:44.751075 27418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:44.751910 27418 raft_consensus.cc:504] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } }
I20260927 21:02:44.752028 27418 leader_election.cc:302] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acfb174d48f14c2eb11cc70484296dd7; no voters:
I20260927 21:02:44.752223 27418 leader_election.cc:288] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:02:44.752290 27421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:44.752511 27418 sys_catalog.cc:564] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:02:44.752533 27421 raft_consensus.cc:686] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [term 1 LEADER]: Becoming Leader. State: Replica: acfb174d48f14c2eb11cc70484296dd7, State: Running, Role: LEADER
I20260927 21:02:44.752753 27421 consensus_queue.cc:237] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } }
I20260927 21:02:44.753182 27423 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acfb174d48f14c2eb11cc70484296dd7. Latest consensus state: current_term: 1 leader_uuid: "acfb174d48f14c2eb11cc70484296dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } } }
I20260927 21:02:44.753266 27423 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:44.753492 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27350
I20260927 21:02:44.753585 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:02:44.753598 27422 sys_catalog.cc:455] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acfb174d48f14c2eb11cc70484296dd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acfb174d48f14c2eb11cc70484296dd7" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45801 } } }
I20260927 21:02:44.753671 27430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:02:44.753687 27422 sys_catalog.cc:458] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7 [sys.catalog]: This master's current role is: LEADER
I20260927 21:02:44.753798 17147 external_mini_cluster.cc:1428] Setting key a5d89b18177be3eef450d66b5fc80737
I20260927 21:02:44.754276 27430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:02:44.755955 27430 catalog_manager.cc:1347] Generated new cluster ID: b251180cd5f94628a3022c17e8d2a6fa
I20260927 21:02:44.756006 27430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:02:44 dist-test-slave-6c7s krb5kdc[27337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:44.764552 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:44.754533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.16.190.254:45801)
0927 21:02:44.754682 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:44.754694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:44.754920 (+ 226us) server_negotiation.cc:407] Connection header received
0927 21:02:44.755082 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:44.755089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:44.755168 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:44.755319 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:44.756031 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.757098 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.757881 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.758329 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.758474 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:44.760994 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:44.761000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:44.761001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:44.761017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:44.762902 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:44.763304 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:44.763308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:44.763308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:44.763422 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:44.763659 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:44.763660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:44.763661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:44.763925 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:44.764102 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:44.764185 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:02:44.764337 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":83,"threads_started":1}
I20260927 21:02:44.770139 27430 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:02:44.770859 27430 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:02:44.778970 27430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: Generated new TSK 0
I20260927 21:02:44.779273 27430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:02:44.812872 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:44.910233 27444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:44.910442 27444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:44.910467 27444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:44.911648 27444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:44.911697 27444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:44.911725 27444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:02:44.913237 27444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:44.913486 27444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:44.913739 27444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:44.919333 27449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:44.919339 27450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:44.919337 27452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:44.919726 27444 server_base.cc:1034] running on GCE node
I20260927 21:02:44.919962 27444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:44.920214 27444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:44.921356 27444 hybrid_clock.cc:648] HybridClock initialized: now 1790542964921344 us; error 30 us; skew 500 ppm
Sep 27 21:02:44 dist-test-slave-6c7s krb5kdc[27337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542964, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:44.923882 27444 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:02:44.924377 27444 webserver.cc:460] Webserver started at http://127.16.190.193:41707/ using document root <none> and password file <none>
I20260927 21:02:44.924608 27444 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:44.924676 27444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:44.924878 27444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:44.925968 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "8e4732848ae248eb896f2fbeeaba1467"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "bc519517a78c4374510b92b6137364fb"
server_key_iv: "80349006ca2093426c6301a512d9b41f"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.926347 27444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "8e4732848ae248eb896f2fbeeaba1467"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "bc519517a78c4374510b92b6137364fb"
server_key_iv: "80349006ca2093426c6301a512d9b41f"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.927701 27444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:02:44.928519 27459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:44.928759 27444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:44.928890 27444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "8e4732848ae248eb896f2fbeeaba1467"
format_stamp: "Formatted at 2026-09-27 21:02:44 on dist-test-slave-6c7s"
server_key: "bc519517a78c4374510b92b6137364fb"
server_key_iv: "80349006ca2093426c6301a512d9b41f"
server_key_version: "encryptionkey@0"
I20260927 21:02:44.928969 27444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:44.946229 27444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:44.948449 27444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:44.948630 27444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:44.948956 27444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:44.949314 27444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:44.949371 27444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:44.949407 27444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:44.949435 27444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:44.955936 27444 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42107
I20260927 21:02:44.955976 27572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42107 every 8 connection(s)
I20260927 21:02:44.956336 27444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:02:44.958900 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27444
I20260927 21:02:44.959019 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:02:44.959214 17147 external_mini_cluster.cc:1428] Setting key 967bbf3d8da6695e7b21b89c39594ed1
Sep 27 21:02:44 dist-test-slave-6c7s krb5kdc[27337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542964, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:02:44.966944 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:44.957145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:58287 (local address 127.16.190.254:45801)
0927 21:02:44.957249 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:44.957251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:44.957408 (+ 157us) server_negotiation.cc:407] Connection header received
0927 21:02:44.957924 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:44.957925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:44.957934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:44.957996 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:44.959009 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.959700 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.961330 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.961611 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.961647 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:44.963961 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:44.963963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:44.963964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:44.963972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:44.965510 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:44.965943 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:44.965944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:44.965944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:44.966004 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:44.966272 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:44.966273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:44.966273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:44.966393 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:44.966471 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:44.966754 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 21:02:44.966847 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 21:02:44.967186 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:44.957120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45801 (local address 127.16.190.193:58287)
0927 21:02:44.957281 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:44.957322 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:44.957840 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:44.958054 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:44.958057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:44.958645 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:44.958878 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:44.958884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.959795 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.959796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:44.961215 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:44.961218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:44.961724 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:44.961725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:44.961827 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:44.962419 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:44.962427 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:44.963799 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:44.965612 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:44.965619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:44.965621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:44.965850 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:44.966084 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:44.966085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:44.966085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:44.966204 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:44.966461 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:44.966467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:44.966616 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 21:02:44.966731 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 21:02:44.966895 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260927 21:02:44.967801 27573 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45801
I20260927 21:02:44.967914 27573 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:44.968119 27573 heartbeater.cc:511] Master 127.16.190.254:45801 requested a full tablet report, sending...
I20260927 21:02:44.968696 27382 ts_manager.cc:194] Registered new tserver with Master: 8e4732848ae248eb896f2fbeeaba1467 (127.16.190.193:42107)
I20260927 21:02:44.969491 27382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:58287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:02:44.988762 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:45.073408 27578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:45.073625 27578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:45.073657 27578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:45.074802 27578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:45.074843 27578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:45.074869 27578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:02:45.076295 27578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:45.076493 27578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:45.076689 27578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:45.082039 27586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:45.082159 27584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:45.082231 27583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:45.082312 27578 server_base.cc:1034] running on GCE node
I20260927 21:02:45.082535 27578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:45.082796 27578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:45.083937 27578 hybrid_clock.cc:648] HybridClock initialized: now 1790542965083922 us; error 43 us; skew 500 ppm
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542965, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:45.086671 27578 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:02:45.087182 27578 webserver.cc:460] Webserver started at http://127.16.190.194:37777/ using document root <none> and password file <none>
I20260927 21:02:45.087422 27578 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:45.087469 27578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:45.087598 27578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:45.088687 27578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "e99e2934ec7742ee95779c8f6b53193d"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "9d5626fb5f57762854376a4465d6b7ff"
server_key_iv: "49491cf16728663c714ef9f5a44d4572"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.089113 27578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "e99e2934ec7742ee95779c8f6b53193d"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "9d5626fb5f57762854376a4465d6b7ff"
server_key_iv: "49491cf16728663c714ef9f5a44d4572"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.090503 27578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:45.091351 27593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.091584 27578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:45.091746 27578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "e99e2934ec7742ee95779c8f6b53193d"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "9d5626fb5f57762854376a4465d6b7ff"
server_key_iv: "49491cf16728663c714ef9f5a44d4572"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.091848 27578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:45.114435 27578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:45.117170 27578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:45.117383 27578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:45.117705 27578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:45.118065 27578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:45.118124 27578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.118163 27578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:45.118196 27578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.124930 27578 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:42433
I20260927 21:02:45.125010 27706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:42433 every 8 connection(s)
I20260927 21:02:45.125352 27578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542965, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:45.134373 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27578
I20260927 21:02:45.134508 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:02:45.134779 17147 external_mini_cluster.cc:1428] Setting key b77c0cd1757d5c027e1d406e4ffc9dd5
I20260927 21:02:45.135908 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.126138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:38909 (local address 127.16.190.254:45801)
0927 21:02:45.126226 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.126228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.126423 (+ 195us) server_negotiation.cc:407] Connection header received
0927 21:02:45.127029 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.127030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.127044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.127101 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:45.128137 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.128657 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.129745 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.129960 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.130052 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.132389 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:45.132390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:45.132391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:45.132399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:45.134049 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.134485 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.134486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.134486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.134549 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.134941 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.134942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.134942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.135097 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:45.135194 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.135767 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.135844 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:02:45.136246 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.126099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45801 (local address 127.16.190.194:38909)
0927 21:02:45.126296 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.126336 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.126882 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.127186 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.127189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.127800 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:45.128021 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.128026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.128767 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.128768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.129650 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.129652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.130064 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.130065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.130195 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.130789 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:45.130798 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:45.132295 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:45.134153 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:45.134161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:45.134163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:45.134397 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:45.134662 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:45.134663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:45.134663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:45.134850 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:45.135414 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:45.135422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:45.135589 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.135731 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.135927 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260927 21:02:45.136880 27707 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45801
I20260927 21:02:45.136994 27707 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:45.137224 27707 heartbeater.cc:511] Master 127.16.190.254:45801 requested a full tablet report, sending...
I20260927 21:02:45.137684 27382 ts_manager.cc:194] Registered new tserver with Master: e99e2934ec7742ee95779c8f6b53193d (127.16.190.194:42433)
I20260927 21:02:45.138270 27382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:38909
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:02:45.164803 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:02:45.247612 27712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:02:45.247843 27712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:02:45.247870 27712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:02:45.249200 27712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:02:45.249253 27712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:02:45.249281 27712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:02:45.250824 27712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:02:45.251083 27712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:02:45.251336 27712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:02:45.257124 27717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:45.257157 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:02:45.257133 27718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:02:45.257395 27712 server_base.cc:1034] running on GCE node
I20260927 21:02:45.257709 27712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:02:45.257998 27712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:02:45.259174 27712 hybrid_clock.cc:648] HybridClock initialized: now 1790542965259163 us; error 31 us; skew 500 ppm
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542965, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:02:45.262005 27712 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:02:45.262512 27712 webserver.cc:460] Webserver started at http://127.16.190.195:33285/ using document root <none> and password file <none>
I20260927 21:02:45.262765 27712 fs_manager.cc:359] Metadata directory not provided
I20260927 21:02:45.262840 27712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:02:45.262993 27712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:02:45.264099 27712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "2477d5de3e364cc1a8b103b102c9b05f"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "836088b2e35b85d760efc4a55fdb7fe6"
server_key_iv: "c8e27fc2425cb7c61fac7543f6b83c4f"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.264482 27712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "2477d5de3e364cc1a8b103b102c9b05f"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "836088b2e35b85d760efc4a55fdb7fe6"
server_key_iv: "c8e27fc2425cb7c61fac7543f6b83c4f"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.265908 27712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:02:45.266686 27727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.266873 27712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:02:45.266978 27712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "2477d5de3e364cc1a8b103b102c9b05f"
format_stamp: "Formatted at 2026-09-27 21:02:45 on dist-test-slave-6c7s"
server_key: "836088b2e35b85d760efc4a55fdb7fe6"
server_key_iv: "c8e27fc2425cb7c61fac7543f6b83c4f"
server_key_version: "encryptionkey@0"
I20260927 21:02:45.267055 27712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:02:45.289016 27712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:02:45.291462 27712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:02:45.291714 27712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:02:45.291980 27712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:02:45.292304 27712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:02:45.292358 27712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.292392 27712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:02:45.292434 27712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:02:45.298671 27712 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:37855
I20260927 21:02:45.298712 27840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:37855 every 8 connection(s)
I20260927 21:02:45.299041 27712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:02:45.300282 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 27712
I20260927 21:02:45.300384 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:02:45.300575 17147 external_mini_cluster.cc:1428] Setting key a94aa298c971affd4ac5ee8f75f155cc
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542965, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:02:45.308562 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.299852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:50101 (local address 127.16.190.254:45801)
0927 21:02:45.299925 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.299927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.300020 (+ 93us) server_negotiation.cc:407] Connection header received
0927 21:02:45.300575 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.300577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.300592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.300673 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:45.301934 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.302431 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.303395 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.303612 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.303668 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.305708 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:45.305709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:45.305709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:45.305716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:45.307221 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.307675 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.307676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.307676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.307730 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.307968 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.307968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.307969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.308087 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:45.308140 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.308419 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.308487 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:02:45.308789 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.299732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45801 (local address 127.16.190.195:50101)
0927 21:02:45.299888 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.299929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.300458 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.300868 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.300873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.301636 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:02:45.301844 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.301849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.302516 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.302517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.303317 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.303318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.303687 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.303688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.303785 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.304262 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:02:45.304269 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:02:45.305614 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:02:45.307304 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:45.307310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:45.307312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:45.307591 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:45.307796 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:02:45.307797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:02:45.307797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:02:45.307901 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:02:45.308150 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:02:45.308155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:02:45.308274 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.308364 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.308484 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260927 21:02:45.309425 27841 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45801
I20260927 21:02:45.309509 27841 heartbeater.cc:464] Registering TS with master...
I20260927 21:02:45.309662 27841 heartbeater.cc:511] Master 127.16.190.254:45801 requested a full tablet report, sending...
I20260927 21:02:45.309995 27382 ts_manager.cc:194] Registered new tserver with Master: 2477d5de3e364cc1a8b103b102c9b05f (127.16.190.195:37855)
I20260927 21:02:45.310330 27382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:50101
I20260927 21:02:45.312394 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:02:45.320108 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.313840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.16.190.254:45801)
0927 21:02:45.313938 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.313940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.314044 (+ 104us) server_negotiation.cc:407] Connection header received
0927 21:02:45.314112 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.314113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.314123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.314200 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:02:45.314780 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.315269 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.315987 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.316180 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.316287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.317517 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:02:45.317518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:02:45.317518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:02:45.317526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:02:45.318866 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.319240 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.319242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.319242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.319308 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:02:45.319514 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:02:45.319515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:02:45.319515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:02:45.319678 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:02:45.319790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.319921 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.320003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:02:45.322007 27382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:02:45.323553 27382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:02:45.330214 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.326057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.16.190.193:42107)
0927 21:02:45.326275 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.326281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.326419 (+ 138us) server_negotiation.cc:407] Connection header received
0927 21:02:45.326483 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.326486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.326505 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.326591 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:45.326858 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.328005 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329218 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.329761 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.329910 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.330073 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.330112 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260927 21:02:45.330215 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.325969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42107 (local address 127.0.0.1:53680)
0927 21:02:45.326326 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.326336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.326402 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.326613 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.326616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.326618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:45.326765 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.326767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.328140 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.328141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.329134 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.329137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.329859 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.329946 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.329961 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.330055 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.330111 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":36,"thread_start_us":92,"threads_started":1}
I20260927 21:02:45.330443 27852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.325957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:37855 (local address 127.0.0.1:45392)
0927 21:02:45.326063 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.326101 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.326195 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.326864 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.326865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.326867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:45.326922 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.326923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.327770 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.327771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.329016 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.329018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.330003 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.330004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.330068 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.330072 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.330118 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.330160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":63,"threads_started":1}
I20260927 21:02:45.330618 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.326324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.16.190.195:37855)
0927 21:02:45.326650 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.326654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.326662 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:02:45.326697 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.326699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.326714 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.326786 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:45.326979 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.327658 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329105 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.329841 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329916 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.329967 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.330251 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.330459 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":80,"threads_started":1}
I20260927 21:02:45.330750 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.326312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.16.190.194:42433)
0927 21:02:45.326536 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.326541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.326560 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:02:45.326715 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.326719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.326736 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.326813 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:45.327005 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.328054 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.329380 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.330049 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.330129 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.330176 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.330610 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.330662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
I20260927 21:02:45.330961 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.326171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:42433 (local address 127.0.0.1:53816)
0927 21:02:45.326485 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.326495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.326538 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.326890 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.326891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.326894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:45.326930 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.326932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.328154 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.328155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.329294 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.329297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.330207 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.330208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.330263 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.330266 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.330839 (+ 573us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.330888 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260927 21:02:45.331038 27507 tablet_service.cc:1467] Processing CreateTablet for tablet 61099cabb5c445028df4542c16657593 (DEFAULT_TABLE table=test-table [id=e2a716ecfdd643d084e901ba0574f1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:45.331391 27507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61099cabb5c445028df4542c16657593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:45.331331 27775 tablet_service.cc:1467] Processing CreateTablet for tablet 61099cabb5c445028df4542c16657593 (DEFAULT_TABLE table=test-table [id=e2a716ecfdd643d084e901ba0574f1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:45.331670 27775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61099cabb5c445028df4542c16657593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:45.333325 27641 tablet_service.cc:1467] Processing CreateTablet for tablet 61099cabb5c445028df4542c16657593 (DEFAULT_TABLE table=test-table [id=e2a716ecfdd643d084e901ba0574f1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:02:45.333691 27641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61099cabb5c445028df4542c16657593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:02:45.334564 27858 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Bootstrap starting.
I20260927 21:02:45.334961 27859 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Bootstrap starting.
I20260927 21:02:45.335300 27858 tablet_bootstrap.cc:654] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:45.335811 27859 tablet_bootstrap.cc:654] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:45.335891 27858 log.cc:824] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:45.336452 27859 log.cc:824] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:45.336679 27861 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Bootstrap starting.
I20260927 21:02:45.336747 27858 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: No bootstrap required, opened a new log
I20260927 21:02:45.336830 27858 ts_tablet_manager.cc:1397] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:45.337355 27861 tablet_bootstrap.cc:654] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Neither blocks nor log segments found. Creating new log.
I20260927 21:02:45.337837 27861 log.cc:824] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:02:45.338516 27861 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: No bootstrap required, opened a new log
I20260927 21:02:45.338577 27861 ts_tablet_manager.cc:1397] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:02:45.338454 27858 raft_consensus.cc:348] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.338636 27858 raft_consensus.cc:374] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:45.338711 27858 raft_consensus.cc:729] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4732848ae248eb896f2fbeeaba1467, State: Initialized, Role: FOLLOWER
I20260927 21:02:45.338837 27858 consensus_queue.cc:260] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.339149 27858 ts_tablet_manager.cc:1428] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:02:45.339178 27573 heartbeater.cc:503] Master 127.16.190.254:45801 was elected leader, sending a full tablet report...
I20260927 21:02:45.339784 27859 tablet_bootstrap.cc:492] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: No bootstrap required, opened a new log
I20260927 21:02:45.339856 27859 ts_tablet_manager.cc:1397] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 21:02:45.340590 27861 raft_consensus.cc:348] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.340725 27861 raft_consensus.cc:374] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:45.340746 27861 raft_consensus.cc:729] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99e2934ec7742ee95779c8f6b53193d, State: Initialized, Role: FOLLOWER
I20260927 21:02:45.340915 27861 consensus_queue.cc:260] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.341195 27861 ts_tablet_manager.cc:1428] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 21:02:45.341221 27707 heartbeater.cc:503] Master 127.16.190.254:45801 was elected leader, sending a full tablet report...
I20260927 21:02:45.341929 27859 raft_consensus.cc:348] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.342128 27859 raft_consensus.cc:374] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:02:45.342175 27859 raft_consensus.cc:729] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2477d5de3e364cc1a8b103b102c9b05f, State: Initialized, Role: FOLLOWER
I20260927 21:02:45.342298 27859 consensus_queue.cc:260] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.342547 27859 ts_tablet_manager.cc:1428] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:02:45.342625 27841 heartbeater.cc:503] Master 127.16.190.254:45801 was elected leader, sending a full tablet report...
W20260927 21:02:45.376037 27708 tablet.cc:2369] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:02:45.457130 27574 tablet.cc:2369] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:02:45.549736 27842 tablet.cc:2369] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:02:45.576365 27866 raft_consensus.cc:482] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:02:45.576524 27866 raft_consensus.cc:504] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.576910 27866 leader_election.cc:288] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e4732848ae248eb896f2fbeeaba1467 (127.16.190.193:42107), e99e2934ec7742ee95779c8f6b53193d (127.16.190.194:42433)
I20260927 21:02:45.581300 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.577412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:52831 (local address 127.16.190.194:42433)
0927 21:02:45.577473 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.577475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.577638 (+ 163us) server_negotiation.cc:407] Connection header received
0927 21:02:45.577676 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.577677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.577686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.577737 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:45.577895 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.578628 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.580278 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.580929 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.580994 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.581001 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.581184 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.581228 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:02:45.581300 27854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.577403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:36367 (local address 127.16.190.193:42107)
0927 21:02:45.577485 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.577487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.577496 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:02:45.577527 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.577528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.577537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.577598 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:02:45.577805 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.578642 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.580278 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.580889 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.580975 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.580986 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.581185 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.581231 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:02:45.581290 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.577195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42107 (local address 127.16.190.195:36367)
0927 21:02:45.577283 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.577297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.577388 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.577630 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.577631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.577632 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:45.577710 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.577712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.578775 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.578777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.580161 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.580164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.580993 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.580994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.581105 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.581107 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.581160 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.581190 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":77}
I20260927 21:02:45.581297 27867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.577326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:42433 (local address 127.16.190.195:52831)
0927 21:02:45.577513 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.577522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:02:45.577629 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:02:45.577757 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:02:45.577758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:02:45.577760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:02:45.577820 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.577821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.578775 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.578777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.580161 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:02:45.580164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.580999 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.581000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:02:45.581076 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.581082 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:02:45.581146 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:02:45.581190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":78,"thread_start_us":100,"threads_started":1}
I20260927 21:02:45.581717 27527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61099cabb5c445028df4542c16657593" candidate_uuid: "2477d5de3e364cc1a8b103b102c9b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4732848ae248eb896f2fbeeaba1467" is_pre_election: true
I20260927 21:02:45.581717 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61099cabb5c445028df4542c16657593" candidate_uuid: "2477d5de3e364cc1a8b103b102c9b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99e2934ec7742ee95779c8f6b53193d" is_pre_election: true
I20260927 21:02:45.581890 27661 raft_consensus.cc:2454] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2477d5de3e364cc1a8b103b102c9b05f in term 0.
I20260927 21:02:45.581890 27527 raft_consensus.cc:2454] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2477d5de3e364cc1a8b103b102c9b05f in term 0.
I20260927 21:02:45.582137 27728 leader_election.cc:302] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2477d5de3e364cc1a8b103b102c9b05f, e99e2934ec7742ee95779c8f6b53193d; no voters:
I20260927 21:02:45.582278 27866 raft_consensus.cc:2781] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:02:45.582371 27866 raft_consensus.cc:482] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:02:45.582405 27866 raft_consensus.cc:3037] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:45.583187 27866 raft_consensus.cc:504] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.583348 27866 leader_election.cc:288] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [CANDIDATE]: Term 1 election: Requested vote from peers 8e4732848ae248eb896f2fbeeaba1467 (127.16.190.193:42107), e99e2934ec7742ee95779c8f6b53193d (127.16.190.194:42433)
I20260927 21:02:45.583634 27661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61099cabb5c445028df4542c16657593" candidate_uuid: "2477d5de3e364cc1a8b103b102c9b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99e2934ec7742ee95779c8f6b53193d"
I20260927 21:02:45.583634 27527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61099cabb5c445028df4542c16657593" candidate_uuid: "2477d5de3e364cc1a8b103b102c9b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4732848ae248eb896f2fbeeaba1467"
I20260927 21:02:45.583737 27527 raft_consensus.cc:3037] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:45.583737 27661 raft_consensus.cc:3037] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:02:45.584720 27661 raft_consensus.cc:2454] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2477d5de3e364cc1a8b103b102c9b05f in term 1.
I20260927 21:02:45.584715 27527 raft_consensus.cc:2454] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2477d5de3e364cc1a8b103b102c9b05f in term 1.
I20260927 21:02:45.584935 27728 leader_election.cc:302] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2477d5de3e364cc1a8b103b102c9b05f, 8e4732848ae248eb896f2fbeeaba1467; no voters:
I20260927 21:02:45.585057 27866 raft_consensus.cc:2781] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:02:45.585253 27866 raft_consensus.cc:686] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 1 LEADER]: Becoming Leader. State: Replica: 2477d5de3e364cc1a8b103b102c9b05f, State: Running, Role: LEADER
I20260927 21:02:45.585367 27866 consensus_queue.cc:237] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } }
I20260927 21:02:45.586308 27381 catalog_manager.cc:5495] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2477d5de3e364cc1a8b103b102c9b05f (127.16.190.195). New cstate: current_term: 1 leader_uuid: "2477d5de3e364cc1a8b103b102c9b05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2477d5de3e364cc1a8b103b102c9b05f" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 37855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 } health_report { overall_health: UNKNOWN } } }
I20260927 21:02:45.621023 27856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.617757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.16.190.195:37855)
0927 21:02:45.617808 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:02:45.617811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:02:45.617818 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:02:45.617852 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:02:45.617853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:02:45.617871 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:02:45.617945 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:02:45.618129 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.618654 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.619934 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:02:45.620323 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:02:45.620387 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:02:45.620567 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:02:45.620711 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:02:45.620801 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:02:45.620882 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 21:02:45.620906 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:02:45.623123 27661 raft_consensus.cc:1261] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 1 FOLLOWER]: Refusing update from remote peer 2477d5de3e364cc1a8b103b102c9b05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:45.623123 27527 raft_consensus.cc:1261] T 61099cabb5c445028df4542c16657593 P 8e4732848ae248eb896f2fbeeaba1467 [term 1 FOLLOWER]: Refusing update from remote peer 2477d5de3e364cc1a8b103b102c9b05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:02:45.623494 27866 consensus_queue.cc:1035] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99e2934ec7742ee95779c8f6b53193d" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:45.623632 27868 consensus_queue.cc:1035] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4732848ae248eb896f2fbeeaba1467" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:02:45.625808 27871 mvcc.cc:205] Tried to move back new op lower bound from 7334063987189518336 to 7334063987038478336. Current Snapshot: MvccSnapshot[applied={T|T < 7334063987189518336}]
I20260927 21:02:45.625985 27873 mvcc.cc:205] Tried to move back new op lower bound from 7334063987189518336 to 7334063987038478336. Current Snapshot: MvccSnapshot[applied={T|T < 7334063987189518336}]
I20260927 21:02:45.626914 27872 mvcc.cc:205] Tried to move back new op lower bound from 7334063987189518336 to 7334063987038478336. Current Snapshot: MvccSnapshot[applied={T|T < 7334063987189518336 or (T in {7334063987189518336})}]
I20260927 21:02:45.629159 27381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:45.629274 27381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:02:45.630110 27381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acfb174d48f14c2eb11cc70484296dd7: Sending DeleteTablet for 3 replicas of tablet 61099cabb5c445028df4542c16657593
I20260927 21:02:45.630445 27507 tablet_service.cc:1514] Processing DeleteTablet for tablet 61099cabb5c445028df4542c16657593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:45 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:53680
I20260927 21:02:45.630496 27641 tablet_service.cc:1514] Processing DeleteTablet for tablet 61099cabb5c445028df4542c16657593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:45 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:53816
I20260927 21:02:45.630705 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27444
I20260927 21:02:45.630687 27775 tablet_service.cc:1514] Processing DeleteTablet for tablet 61099cabb5c445028df4542c16657593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:02:45 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:45392
I20260927 21:02:45.631348 27882 tablet_replica.cc:330] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: stopping tablet replica
I20260927 21:02:45.631450 27882 raft_consensus.cc:2229] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:02:45.631578 27882 raft_consensus.cc:2258] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:45.632063 27882 ts_tablet_manager.cc:1905] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:45.632550 27881 tablet_replica.cc:330] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: stopping tablet replica
I20260927 21:02:45.632737 27881 raft_consensus.cc:2229] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:02:45.632936 27881 raft_consensus.cc:2258] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:02:45.633523 27881 ts_tablet_manager.cc:1905] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:02:45.633929 27882 ts_tablet_manager.cc:1918] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:45.634066 27882 log.cc:1196] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/61099cabb5c445028df4542c16657593
I20260927 21:02:45.634418 27882 ts_tablet_manager.cc:1939] T 61099cabb5c445028df4542c16657593 P 2477d5de3e364cc1a8b103b102c9b05f: Deleting consensus metadata
I20260927 21:02:45.635489 27881 ts_tablet_manager.cc:1918] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:02:45.635569 27881 log.cc:1196] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/61099cabb5c445028df4542c16657593
I20260927 21:02:45.635870 27881 ts_tablet_manager.cc:1939] T 61099cabb5c445028df4542c16657593 P e99e2934ec7742ee95779c8f6b53193d: Deleting consensus metadata
I20260927 21:02:45.636327 27366 catalog_manager.cc:4841] TS e99e2934ec7742ee95779c8f6b53193d (127.16.190.194:42433): tablet 61099cabb5c445028df4542c16657593 (table test-table [id=e2a716ecfdd643d084e901ba0574f1cb]) successfully deleted
I20260927 21:02:45.636969 27367 catalog_manager.cc:4841] TS 2477d5de3e364cc1a8b103b102c9b05f (127.16.190.195:37855): tablet 61099cabb5c445028df4542c16657593 (table test-table [id=e2a716ecfdd643d084e901ba0574f1cb]) successfully deleted
W20260927 21:02:45.637220 27366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:42107 (error 108)
I20260927 21:02:45.637455 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27578
I20260927 21:02:45.638022 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:02:45.637673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42107 (local address 127.0.0.1:53684)
0927 21:02:45.637938 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 21:02:45.637980 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
W20260927 21:02:45.638190 27366 catalog_manager.cc:4584] TS 8e4732848ae248eb896f2fbeeaba1467 (127.16.190.193:42107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61099cabb5c445028df4542c16657593: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42107: connect: Connection refused (error 111)
I20260927 21:02:45.643294 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27712
I20260927 21:02:45.648358 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 27350
2026-09-27T21:02:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2207 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27886](info): set up 2 sockets
Sep 27 21:02:45 dist-test-slave-6c7s krb5kdc[27886](info): commencing operation
krb5kdc: starting...
W20260927 21:02:46.702560 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:02:46 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790542966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:02:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangeradmin_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangeradmin_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangerlookup_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangerlookup_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
I20260927 21:02:46.880589 17147 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQTa3tm/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/postgres -l logfile start
2026-09-27 21:02:49.832 UTC [27916] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:02:49.832 UTC [27916] LOG: listening on IPv4 address "127.16.190.212", port 41009
2026-09-27 21:02:49.836 UTC [27916] LOG: listening on Unix socket "/tmp/.s.PGSQL.41009"
2026-09-27 21:02:49.876 UTC [27919] LOG: database system was shut down at 2026-09-27 21:02:48 UTC
2026-09-27 21:02:49.881 UTC [27916] LOG: database system is ready to accept connections
I20260927 21:02:50.858445 17147 mini_postgres.cc:96] Postgres bound to 41009
2026-09-27 21:02:50.865 UTC [27927] FATAL: database "slave" does not exist
127.16.190.212:41009 - accepting connections
I20260927 21:02:50.866258 17147 mini_ranger.cc:161] Starting Ranger...
I20260927 21:02:50.882751 17147 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:02:51.480389 17147 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:02:51.480526 17147 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin
2026-09-27 21:02:51,850 [I] DB FLAVOR :POSTGRES
2026-09-27 21:02:51,851 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:02:51,851 [I] Checking connection..
2026-09-27 21:02:51,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:02:52,108 [I] Checking connection passed.
2026-09-27 21:02:52,108 [I] --------- Verifying version history table ---------
2026-09-27 21:02:52,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:02:52,366 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:02:52,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:02:52,604 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:02:52,605 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:02:52,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:02:52.846 UTC [27991] WARNING: there is no transaction in progress
2026-09-27 21:02:52,853 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:02:52,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:02:53,098 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:02:53,098 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:02:53,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:02:53,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:02:53,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6c7s.c.gcp-upstream.internal','N') ;"
2026-09-27 21:02:53,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:02:54,096 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:02:54,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:02:54,333 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:02:54,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:02:54,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:02:54,868 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:02:54,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:02:55.908 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:55.926 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:55.945 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:55.961 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:55.978 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:55.997 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.061 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.069 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.086 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.097 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.108 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.123 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.134 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.142 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.150 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56.158 UTC [28099] WARNING: there is no transaction in progress
2026-09-27 21:02:56,916 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:02:56,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:02:57,164 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:02:57,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:02:57,400 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:02:57,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:02:57,636 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:02:57,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:02:57,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:02:58,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6c7s.c.gcp-upstream.internal';"
2026-09-27 21:02:58,399 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:02:58,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:02:58,667 [I] DB_PATCHES have already been applied
I20260927 21:02:58.674707 17147 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQTa3tm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:02:58.674773 17147 mini_ranger.cc:193] Using host: 127.16.190.212
I20260927 21:02:58.677862 17147 mini_ranger.cc:227] Ranger admin URL: http://127.16.190.212:41197
Sep 27, 2026 9:02:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:02:59.805557 17147 mini_ranger.cc:160] Time spent starting Ranger: real 8.939s user 0.001s sys 0.008s
Sep 27, 2026 9:03:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:03:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 9:03:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:03:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:03:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:03:22.405092 17147 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangerkms_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/rangerkms_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.190.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.190.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:03:22 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543002, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:03:22.514549 17147 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:03:22.525159 17147 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:03:23.562536 17147 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:03:23.562633 17147 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms
I20260927 21:03:23.562654 17147 mini_ranger_kms.cc:227] Using postgres at 127.16.190.212:41009
2026-09-27 21:03:23,775 [I] DB FLAVOR :POSTGRES
2026-09-27 21:03:23,776 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:03:23,776 [I] Checking connection
2026-09-27 21:03:23,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:03:24,040 [I] connection success
2026-09-27 21:03:24,040 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:03:24,041 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:03:24,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:03:24,281 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:03:24,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:03:24,281 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:03:24,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:41009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:03:24,558 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:03:25.000054 17147 mini_ranger_kms.cc:310] Created kms service
I20260927 21:03:25.222712 17147 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:03:25.267941 17147 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:03:25 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543002, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:25.700153 17147 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:03:25.700481 17147 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:03:25.700512 17147 mini_ranger_kms.cc:240] Using host: 127.16.190.212
I20260927 21:03:25.704598 17147 mini_ranger_kms.cc:276] Ranger KMS PID: 28280
I20260927 21:03:25.704712 17147 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.190.212:45507
Sep 27, 2026 9:03:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:03:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:03:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:03:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:03:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:03:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 21:03:27.943233 17147 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.429s user 0.011s sys 0.183s
I20260927 21:03:27.943369 17147 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:03:27.943460 17147 mini_ranger_kms.cc:401] 127.16.190.212:45507/kms/v1/keys
Sep 27 21:03:28 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 21:03:28 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
Sep 27 21:03:28 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
Sep 27 21:03:31 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:03:31.324707 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:34513
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:34513
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:45507/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:03:31.406481 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:03:31.406749 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:03:31.406785 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:03:31.407981 28330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:03:31.408064 28330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:03:31.408089 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:03:31.408109 28330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:03:31.408141 28330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:03:31.409799 28330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.190.212:45507/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:34513
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:03:31.410092 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:03:31.410326 28330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:03:31.415424 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:03:31.415448 28330 server_base.cc:1034] running on GCE node
W20260927 21:03:31.415560 28335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:31.415503 28338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:03:31.415935 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:03:31.416208 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:03:31.417342 28330 hybrid_clock.cc:648] HybridClock initialized: now 1790543011417324 us; error 36 us; skew 500 ppm
Sep 27 21:03:31 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:03:31.419970 28330 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:03:31.420434 28330 webserver.cc:460] Webserver started at http://127.16.190.254:35989/ using document root <none> and password file <none>
I20260927 21:03:31.420621 28330 fs_manager.cc:359] Metadata directory not provided
I20260927 21:03:31.420658 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:03:31.420748 28330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:03:31 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:31.474115 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "1f5f3df8e22a49d2a875571547ca3090"
format_stamp: "Formatted at 2026-09-27 21:03:31 on dist-test-slave-6c7s"
server_key: "6c9f3c3c2860dc9c0f2664e9ba58ed53"
server_key_iv: "34df32e31d94b40ac53b4c187aafb98c"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:31.474524 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "1f5f3df8e22a49d2a875571547ca3090"
format_stamp: "Formatted at 2026-09-27 21:03:31 on dist-test-slave-6c7s"
server_key: "6c9f3c3c2860dc9c0f2664e9ba58ed53"
server_key_iv: "34df32e31d94b40ac53b4c187aafb98c"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:31.476013 28330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:03:31.491922 28345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:31.492159 28330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:03:31.492262 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "1f5f3df8e22a49d2a875571547ca3090"
format_stamp: "Formatted at 2026-09-27 21:03:31 on dist-test-slave-6c7s"
server_key: "6c9f3c3c2860dc9c0f2664e9ba58ed53"
server_key_iv: "34df32e31d94b40ac53b4c187aafb98c"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:31.492377 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:03:31.511739 28330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:03:31.513983 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:03:31.514176 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:03:31.519256 28330 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:34513
I20260927 21:03:31.519280 28397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:34513 every 8 connection(s)
I20260927 21:03:31.519708 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:03:31.520385 28398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:03:31.521029 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 28330
I20260927 21:03:31.521159 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:03:31.521382 17147 external_mini_cluster.cc:1428] Setting key 46b51616024af6b6250c4ec39072c779
I20260927 21:03:31.523360 28398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: Bootstrap starting.
I20260927 21:03:31.524082 28398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: Neither blocks nor log segments found. Creating new log.
I20260927 21:03:31.524415 28398 log.cc:824] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: Log is configured to *not* fsync() on all Append() calls
I20260927 21:03:31.525301 28398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: No bootstrap required, opened a new log
I20260927 21:03:31.527279 28398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } }
I20260927 21:03:31.527464 28398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:03:31.527503 28398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5f3df8e22a49d2a875571547ca3090, State: Initialized, Role: FOLLOWER
I20260927 21:03:31.527684 28398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } }
I20260927 21:03:31.527766 28398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:03:31.527794 28398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:03:31.527856 28398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:03:31.528514 28398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } }
I20260927 21:03:31.528641 28398 leader_election.cc:302] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f5f3df8e22a49d2a875571547ca3090; no voters:
I20260927 21:03:31.528853 28398 leader_election.cc:288] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:03:31.528904 28403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:03:31.529150 28403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [term 1 LEADER]: Becoming Leader. State: Replica: 1f5f3df8e22a49d2a875571547ca3090, State: Running, Role: LEADER
I20260927 21:03:31.529258 28398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:03:31.529281 28403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } }
I20260927 21:03:31.529708 28403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f5f3df8e22a49d2a875571547ca3090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } } }
I20260927 21:03:31.529870 28403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [sys.catalog]: This master's current role is: LEADER
I20260927 21:03:31.530051 28403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f5f3df8e22a49d2a875571547ca3090. Latest consensus state: current_term: 1 leader_uuid: "1f5f3df8e22a49d2a875571547ca3090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f3df8e22a49d2a875571547ca3090" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 34513 } } }
I20260927 21:03:31.530167 28403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090 [sys.catalog]: This master's current role is: LEADER
I20260927 21:03:31.530323 28412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:03:31.530887 28412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:03:31.531333 28398 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 27 21:03:31 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:03:31.538089 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:31.521997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.16.190.254:34513)
0927 21:03:31.522191 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:31.522202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:31.523725 (+ 1523us) server_negotiation.cc:407] Connection header received
0927 21:03:31.524343 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:31.524353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:31.524442 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:31.524594 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:03:31.525556 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:31.527013 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:31.528132 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:31.528786 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:31.528930 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:31.532002 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:03:31.532006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:03:31.532007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:03:31.532020 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:03:31.534140 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:31.534567 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:31.534570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:31.534570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:31.534667 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:31.534864 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:31.534865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:31.534865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:31.535177 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:03:31.535346 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:31.535808 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 21:03:31.535929 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 21:03:31.541517 28412 catalog_manager.cc:1347] Generated new cluster ID: 5c2869ed16e54d3d9edd41cb33c24ef8
I20260927 21:03:31.541592 28412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:03:31.550133 28412 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:03:31.550997 28412 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:03:31.559816 28412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: Generated new TSK 0
I20260927 21:03:31.560122 28412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:03:32.277824 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:45507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:03:32.366457 28443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:03:32.366712 28443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:03:32.366753 28443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:03:32.368017 28443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:03:32.368158 28443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:03:32.368233 28443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:03:32.369995 28443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.190.212:45507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:03:32.370271 28443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:03:32.370563 28443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:03:32.375881 28451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.376048 28449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.375881 28448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:03:32.375955 28443 server_base.cc:1034] running on GCE node
I20260927 21:03:32.376433 28443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:03:32.376726 28443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:03:32.377928 28443 hybrid_clock.cc:648] HybridClock initialized: now 1790543012377917 us; error 43 us; skew 500 ppm
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:03:32.381006 28443 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:03:32.381454 28443 webserver.cc:460] Webserver started at http://127.16.190.193:40085/ using document root <none> and password file <none>
I20260927 21:03:32.381690 28443 fs_manager.cc:359] Metadata directory not provided
I20260927 21:03:32.381758 28443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:03:32.381893 28443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:32.399012 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "74ccb4a3fbb52f3f39324e533e76a585"
server_key_iv: "2885ea65c3455cf37b5bf315496446e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.399525 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "74ccb4a3fbb52f3f39324e533e76a585"
server_key_iv: "2885ea65c3455cf37b5bf315496446e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.401307 28443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 21:03:32.421428 28458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.421700 28443 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260927 21:03:32.421793 28443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "74ccb4a3fbb52f3f39324e533e76a585"
server_key_iv: "2885ea65c3455cf37b5bf315496446e3"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.421895 28443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:03:32.465176 28443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:03:32.467809 28443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:03:32.468001 28443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:03:32.468257 28443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:03:32.468673 28443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:03:32.468727 28443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.468760 28443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:03:32.468776 28443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.476526 28443 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:33773
I20260927 21:03:32.476606 28573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:33773 every 8 connection(s)
I20260927 21:03:32.477080 28443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:03:32.477334 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 28443
I20260927 21:03:32.477433 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:03:32.477640 17147 external_mini_cluster.cc:1428] Setting key 5ee69e89d19f051513186479145c8faf
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:03:32.489950 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:32.478504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:57385 (local address 127.16.190.254:34513)
0927 21:03:32.478629 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:32.478632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:32.478826 (+ 194us) server_negotiation.cc:407] Connection header received
0927 21:03:32.479545 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:32.479546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:32.479561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:32.479613 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:03:32.480490 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.481343 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.482845 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.483161 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.483210 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:32.485975 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:03:32.485976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:03:32.485976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:03:32.485986 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:03:32.488024 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:32.488614 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:32.488616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:32.488616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:32.488692 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:32.489146 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:32.489147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:32.489147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:32.489301 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:03:32.489380 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:32.489772 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 21:03:32.489855 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260927 21:03:32.490309 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:32.478524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:34513 (local address 127.16.190.193:57385)
0927 21:03:32.478700 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:32.478749 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:32.479414 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:32.479818 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:32.479824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:32.480098 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:03:32.480376 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:32.480382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.481460 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.481461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:32.482743 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:32.482746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.483317 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.483318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:32.483460 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:32.484146 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:03:32.484157 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:03:32.485834 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:03:32.488137 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:32.488146 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:32.488148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:32.488474 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:32.488796 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:32.488797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:32.488797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:32.489026 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:32.489396 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:03:32.489403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:03:32.489593 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 21:03:32.489730 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 21:03:32.489904 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:03:32.490926 28574 heartbeater.cc:347] Connected to a master server at 127.16.190.254:34513
I20260927 21:03:32.491034 28574 heartbeater.cc:464] Registering TS with master...
I20260927 21:03:32.491243 28574 heartbeater.cc:511] Master 127.16.190.254:34513 requested a full tablet report, sending...
I20260927 21:03:32.491777 28362 ts_manager.cc:194] Registered new tserver with Master: 98411c6cc8c547e3bd7e12c9a1aaf6f9 (127.16.190.193:33773)
I20260927 21:03:32.492400 28362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:57385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:03:32.510624 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:45507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 21:03:32.642931 28580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:03:32.643314 28580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:03:32.643407 28580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:03:32.645596 28580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:03:32.645740 28580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:03:32.645855 28580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:03:32.648558 28580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.190.212:45507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:03:32.649263 28580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:03:32.649629 28580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:03:32.656669 28590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.656687 28587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.656967 28588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:03:32.657034 28580 server_base.cc:1034] running on GCE node
I20260927 21:03:32.657320 28580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:03:32.657711 28580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:03:32.658906 28580 hybrid_clock.cc:648] HybridClock initialized: now 1790543012658879 us; error 36 us; skew 500 ppm
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:03:32.662436 28580 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:03:32.663106 28580 webserver.cc:460] Webserver started at http://127.16.190.194:40227/ using document root <none> and password file <none>
I20260927 21:03:32.663388 28580 fs_manager.cc:359] Metadata directory not provided
I20260927 21:03:32.663444 28580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:03:32.663631 28580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:32.677515 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "c0b95448a08d4aef93071488929cf25e"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "6a82248b24441ef3dbf32a1349b8d8cf"
server_key_iv: "867af65cffa448d9ebe1068f13dfe64a"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.678695 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "c0b95448a08d4aef93071488929cf25e"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "6a82248b24441ef3dbf32a1349b8d8cf"
server_key_iv: "867af65cffa448d9ebe1068f13dfe64a"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.681676 28580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:03:32.695793 28598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.696139 28580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:03:32.696247 28580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "c0b95448a08d4aef93071488929cf25e"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "6a82248b24441ef3dbf32a1349b8d8cf"
server_key_iv: "867af65cffa448d9ebe1068f13dfe64a"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.696353 28580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:03:32.723795 28580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:03:32.726502 28580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:03:32.726718 28580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:03:32.727039 28580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:03:32.727496 28580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:03:32.727592 28580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.727672 28580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:03:32.727737 28580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.737041 28580 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:39845
I20260927 21:03:32.737567 28580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:03:32.738176 28711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:39845 every 8 connection(s)
I20260927 21:03:32.741405 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 28580
I20260927 21:03:32.741515 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:03:32.741729 17147 external_mini_cluster.cc:1428] Setting key 40a80ea10e6e34d9f1d900396392f2e5
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:03:32.749523 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:32.738364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:43141 (local address 127.16.190.254:34513)
0927 21:03:32.738432 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:32.738434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:32.738682 (+ 248us) server_negotiation.cc:407] Connection header received
0927 21:03:32.739354 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:32.739355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:32.739373 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:32.739469 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:03:32.740289 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.741184 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.742742 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.743046 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.743097 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:32.745909 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:03:32.745911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:03:32.745911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:03:32.745920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:03:32.747716 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:32.748279 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:32.748281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:32.748281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:32.748345 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:32.748679 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:32.748680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:32.748681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:32.748859 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:03:32.748911 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:32.749309 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 21:03:32.749424 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:03:32.750056 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:32.738369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:34513 (local address 127.16.190.194:43141)
0927 21:03:32.738530 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:32.738573 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:32.739223 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:32.739576 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:32.739580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:32.739841 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:03:32.740160 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:32.740168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.741307 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.741308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:32.742627 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:32.742631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:32.743201 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:32.743202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:32.743352 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:32.744028 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:03:32.744039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:03:32.745776 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:03:32.747830 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:32.747839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:32.747841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:32.748177 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:32.748438 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:32.748439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:32.748439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:32.748591 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:32.748963 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:03:32.748968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:03:32.749139 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 21:03:32.749630 (+ 491us) client_negotiation.cc:240] Negotiation successful
0927 21:03:32.749809 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260927 21:03:32.750726 28712 heartbeater.cc:347] Connected to a master server at 127.16.190.254:34513
I20260927 21:03:32.750885 28712 heartbeater.cc:464] Registering TS with master...
I20260927 21:03:32.751143 28712 heartbeater.cc:511] Master 127.16.190.254:34513 requested a full tablet report, sending...
I20260927 21:03:32.751590 28362 ts_manager.cc:194] Registered new tserver with Master: c0b95448a08d4aef93071488929cf25e (127.16.190.194:39845)
I20260927 21:03:32.752072 28362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:43141
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:03:32.776659 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:45507/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:03:32.905467 28717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:03:32.905853 28717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:03:32.905944 28717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:03:32.907797 28717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:03:32.907876 28717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:03:32.907919 28717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:03:32.910229 28717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.190.212:45507/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:03:32.910535 28717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:03:32.910970 28717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:03:32.917553 28723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.917554 28722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:03:32.918026 28725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:03:32.918900 28717 server_base.cc:1034] running on GCE node
I20260927 21:03:32.919180 28717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:03:32.919495 28717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:03:32.922807 28717 hybrid_clock.cc:648] HybridClock initialized: now 1790543012922787 us; error 48 us; skew 500 ppm
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:03:32.929108 28717 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:03:32.929728 28717 webserver.cc:460] Webserver started at http://127.16.190.195:42637/ using document root <none> and password file <none>
I20260927 21:03:32.930001 28717 fs_manager.cc:359] Metadata directory not provided
I20260927 21:03:32.930076 28717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:03:32.930240 28717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:03:32 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:32.956144 28717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "c65b4d97b96d4ceb855da0028f5ad754"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "65e62c941d60b53b2a8ca3351da67776"
server_key_iv: "1cce855bacc18214371029d3ae9928b8"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.957672 28717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "c65b4d97b96d4ceb855da0028f5ad754"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "65e62c941d60b53b2a8ca3351da67776"
server_key_iv: "1cce855bacc18214371029d3ae9928b8"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.963692 28717 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260927 21:03:32.985030 28732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:32.985296 28717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 21:03:32.985437 28717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "c65b4d97b96d4ceb855da0028f5ad754"
format_stamp: "Formatted at 2026-09-27 21:03:32 on dist-test-slave-6c7s"
server_key: "65e62c941d60b53b2a8ca3351da67776"
server_key_iv: "1cce855bacc18214371029d3ae9928b8"
server_key_version: "kuduclusterkey@0"
I20260927 21:03:32.985623 28717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:03:33.003415 28717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:03:33.012344 28717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:03:33.012626 28717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:03:33.013180 28717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:03:33.013724 28717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:03:33.013826 28717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:33.013908 28717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:03:33.013973 28717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:03:33.042652 28717 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34555
I20260927 21:03:33.048848 28845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34555 every 8 connection(s)
I20260927 21:03:33.049242 28717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:03:33.054746 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 28717
I20260927 21:03:33.054852 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:03:33.055065 17147 external_mini_cluster.cc:1428] Setting key 4fcc06be374a9f1100a6891f378c5d5c
Sep 27 21:03:33 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:03:33.076386 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.051410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:34513 (local address 127.16.190.195:36271)
0927 21:03:33.051826 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.051873 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.052913 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.053337 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.053342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.053719 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:03:33.054137 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.054146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.057739 (+ 3593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.057740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.059171 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.059175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.069510 (+ 10335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.069511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.069736 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.070398 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:03:33.070414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:03:33.072177 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:03:33.074539 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:33.074550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:33.074553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:33.074894 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:33.075133 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:03:33.075134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:03:33.075134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:03:33.075285 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:03:33.075593 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:03:33.075601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:03:33.075792 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.075920 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.076125 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":149,"threads_started":1}
I20260927 21:03:33.076982 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.051247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:36271 (local address 127.16.190.254:34513)
0927 21:03:33.051323 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.051325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.052008 (+ 683us) server_negotiation.cc:407] Connection header received
0927 21:03:33.053078 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.053080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.053095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.053131 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:03:33.054278 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.057583 (+ 3305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.068952 (+ 11369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.069318 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.069380 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.072317 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:03:33.072319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:03:33.072319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:03:33.072328 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:03:33.074425 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:33.075002 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:33.075004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:33.075004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:33.075066 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:33.075356 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:33.075356 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:33.075357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:33.075500 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:03:33.075565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.076707 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.076863 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:03:33.077642 28846 heartbeater.cc:347] Connected to a master server at 127.16.190.254:34513
I20260927 21:03:33.077760 28846 heartbeater.cc:464] Registering TS with master...
I20260927 21:03:33.077975 28846 heartbeater.cc:511] Master 127.16.190.254:34513 requested a full tablet report, sending...
I20260927 21:03:33.078370 28362 ts_manager.cc:194] Registered new tserver with Master: c65b4d97b96d4ceb855da0028f5ad754 (127.16.190.195:34555)
I20260927 21:03:33.078864 28362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:36271
I20260927 21:03:33.079366 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:03:33.089792 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.081551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.16.190.254:34513)
0927 21:03:33.081593 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.081595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.081916 (+ 321us) server_negotiation.cc:407] Connection header received
0927 21:03:33.082040 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.082041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.082051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.082081 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:03:33.082825 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.083592 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.084661 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.084994 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.085037 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.086678 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:03:33.086679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:03:33.086680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:03:33.086687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:03:33.088436 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:33.088849 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:33.088850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:33.088850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:33.088903 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:03:33.089113 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:03:33.089114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:03:33.089114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:03:33.089237 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:03:33.089326 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.089614 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.089694 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:03:33.092271 28361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:03:33.094554 28361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:03:33.104732 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39845 (local address 127.0.0.1:39738)
0927 21:03:33.099443 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.099451 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.099488 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.099754 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.099757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.099759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:03:33.099826 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.099828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.101459 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.101460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.102935 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.102938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.104414 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.104415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.104501 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.104515 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.104584 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.104638 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":76,"threads_started":1}
I20260927 21:03:33.105222 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.16.190.194:39845)
0927 21:03:33.099185 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.099191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.099538 (+ 347us) server_negotiation.cc:407] Connection header received
0927 21:03:33.099582 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.099584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.099599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.099662 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:03:33.099896 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.101373 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.103415 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.104201 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.104289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.104345 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.105091 (+ 746us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.105135 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":71,"threads_started":1}
I20260927 21:03:33.106284 28634 tablet_service.cc:1467] Processing CreateTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d (DEFAULT_TABLE table=test-table [id=c7881419547247e888e289eca97c6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:03:33.106650 28634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:03:33.110795 28865 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Bootstrap starting.
I20260927 21:03:33.111869 28865 tablet_bootstrap.cc:654] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Neither blocks nor log segments found. Creating new log.
I20260927 21:03:33.112262 28865 log.cc:824] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:03:33.113269 28865 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: No bootstrap required, opened a new log
I20260927 21:03:33.113332 28865 ts_tablet_manager.cc:1397] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:03:33.114310 28859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:33773 (local address 127.0.0.1:57898)
0927 21:03:33.101877 (+ 3384us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.101888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.103326 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.109017 (+ 5691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.109019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.109021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:03:33.109071 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.109073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.110348 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.110349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.111517 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.111521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.114057 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.114058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.114122 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.114126 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.114184 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.114232 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3367,"thread_start_us":54,"threads_started":1}
I20260927 21:03:33.115278 28865 raft_consensus.cc:348] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.115435 28865 raft_consensus.cc:374] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:03:33.115465 28865 raft_consensus.cc:729] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b95448a08d4aef93071488929cf25e, State: Initialized, Role: FOLLOWER
I20260927 21:03:33.115579 28865 consensus_queue.cc:260] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.115828 28865 ts_tablet_manager.cc:1428] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:03:33.116142 28712 heartbeater.cc:503] Master 127.16.190.254:34513 was elected leader, sending a full tablet report...
I20260927 21:03:33.121045 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.16.190.193:33773)
0927 21:03:33.108714 (+ 10135us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.108720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.108734 (+ 14us) server_negotiation.cc:407] Connection header received
0927 21:03:33.108776 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.108779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.108798 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.108925 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:03:33.109162 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.110244 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.113109 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.113881 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.113958 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.114018 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.120893 (+ 6875us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.120959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10090,"thread_start_us":53,"threads_started":1}
I20260927 21:03:33.122352 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d (DEFAULT_TABLE table=test-table [id=c7881419547247e888e289eca97c6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:03:33.122740 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:03:33.123896 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34555 (local address 127.0.0.1:53782)
0927 21:03:33.099247 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.099288 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.099349 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.114520 (+ 15171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.114521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.114524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:03:33.114565 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.114567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.121397 (+ 6830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.121399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.122602 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.122605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.123653 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.123655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.123724 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.123729 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.123762 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.123807 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":68,"threads_started":1}
I20260927 21:03:33.124351 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.098260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.16.190.195:34555)
0927 21:03:33.112928 (+ 14668us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.112934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.112945 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:03:33.112985 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.112987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.113004 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.114646 (+ 1642us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:03:33.114712 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.115750 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.122751 (+ 7001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.123545 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.124111 (+ 566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.124191 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.124220 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.124262 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14625,"thread_start_us":90,"threads_started":1}
I20260927 21:03:33.125031 28780 tablet_service.cc:1467] Processing CreateTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d (DEFAULT_TABLE table=test-table [id=c7881419547247e888e289eca97c6324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:03:33.125378 28780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:03:33.131239 28869 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Bootstrap starting.
Sep 27 21:03:33 dist-test-slave-6c7s krb5kdc[27886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:03:33.132241 28869 tablet_bootstrap.cc:654] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Neither blocks nor log segments found. Creating new log.
I20260927 21:03:33.132550 28869 log.cc:824] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Log is configured to *not* fsync() on all Append() calls
I20260927 21:03:33.137076 28868 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: Bootstrap starting.
I20260927 21:03:33.137897 28868 tablet_bootstrap.cc:654] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: Neither blocks nor log segments found. Creating new log.
I20260927 21:03:33.138212 28868 log.cc:824] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:03:33.139525 28867 raft_consensus.cc:482] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:03:33.139650 28867 raft_consensus.cc:504] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.139885 28869 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: No bootstrap required, opened a new log
I20260927 21:03:33.139966 28869 ts_tablet_manager.cc:1397] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 21:03:33.141944 28869 raft_consensus.cc:348] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.142125 28869 raft_consensus.cc:374] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:03:33.142154 28869 raft_consensus.cc:729] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c65b4d97b96d4ceb855da0028f5ad754, State: Initialized, Role: FOLLOWER
I20260927 21:03:33.142261 28869 consensus_queue.cc:260] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.142513 28869 ts_tablet_manager.cc:1428] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:03:33.142976 28846 heartbeater.cc:503] Master 127.16.190.254:34513 was elected leader, sending a full tablet report...
I20260927 21:03:33.144842 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.140118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34555 (local address 127.16.190.194:49871)
0927 21:03:33.140210 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.140221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.140261 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.140628 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.140629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.140630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:03:33.140700 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.140702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.141644 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.141646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.142847 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.142849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.144555 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.144556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.144648 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.144651 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.144698 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.144744 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 21:03:33.145291 28860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.140336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:49871 (local address 127.16.190.195:34555)
0927 21:03:33.140375 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.140378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.140384 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:03:33.140409 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.140410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.140418 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.140462 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:03:33.140775 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.141546 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.143809 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.144427 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.144500 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.144509 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.145187 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.145234 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:03:33.145565 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aa9eacfd24f5abd2a7f2d76aadc1d" candidate_uuid: "c0b95448a08d4aef93071488929cf25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65b4d97b96d4ceb855da0028f5ad754" is_pre_election: true
I20260927 21:03:33.145714 28796 raft_consensus.cc:2454] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0b95448a08d4aef93071488929cf25e in term 0.
I20260927 21:03:33.146028 28599 leader_election.cc:302] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0b95448a08d4aef93071488929cf25e, c65b4d97b96d4ceb855da0028f5ad754; no voters:
I20260927 21:03:33.146284 28873 raft_consensus.cc:2781] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:03:33.146364 28867 leader_election.cc:288] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c65b4d97b96d4ceb855da0028f5ad754 (127.16.190.195:34555), 98411c6cc8c547e3bd7e12c9a1aaf6f9 (127.16.190.193:33773)
I20260927 21:03:33.146441 28873 raft_consensus.cc:482] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:03:33.146567 28873 raft_consensus.cc:3037] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:03:33.151579 28868 tablet_bootstrap.cc:492] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: No bootstrap required, opened a new log
I20260927 21:03:33.151676 28868 ts_tablet_manager.cc:1397] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260927 21:03:33.151882 28873 raft_consensus.cc:504] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.152243 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aa9eacfd24f5abd2a7f2d76aadc1d" candidate_uuid: "c0b95448a08d4aef93071488929cf25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c65b4d97b96d4ceb855da0028f5ad754"
I20260927 21:03:33.152339 28796 raft_consensus.cc:3037] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:03:33.153296 28796 raft_consensus.cc:2454] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0b95448a08d4aef93071488929cf25e in term 1.
I20260927 21:03:33.153501 28599 leader_election.cc:302] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0b95448a08d4aef93071488929cf25e, c65b4d97b96d4ceb855da0028f5ad754; no voters:
I20260927 21:03:33.153594 28867 raft_consensus.cc:2781] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:03:33.153734 28867 raft_consensus.cc:686] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 1 LEADER]: Becoming Leader. State: Replica: c0b95448a08d4aef93071488929cf25e, State: Running, Role: LEADER
I20260927 21:03:33.153825 28867 consensus_queue.cc:237] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.154515 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.145758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:33773 (local address 127.16.190.194:35215)
0927 21:03:33.146264 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.146275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.146316 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.147147 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:03:33.147148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:03:33.147150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:03:33.147186 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.147187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.149868 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.149869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.150991 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:03:33.150993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.154225 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.154226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:03:33.154300 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.154304 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:03:33.154353 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 21:03:33.154415 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489}
I20260927 21:03:33.154388 28868 raft_consensus.cc:348] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.154572 28868 raft_consensus.cc:374] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:03:33.154603 28868 raft_consensus.cc:729] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98411c6cc8c547e3bd7e12c9a1aaf6f9, State: Initialized, Role: FOLLOWER
I20260927 21:03:33.154729 28868 consensus_queue.cc:260] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } }
I20260927 21:03:33.155018 28868 ts_tablet_manager.cc:1428] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 21:03:33.155038 28574 heartbeater.cc:503] Master 127.16.190.254:34513 was elected leader, sending a full tablet report...
I20260927 21:03:33.154942 28362 catalog_manager.cc:5495] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e reported cstate change: term changed from 0 to 1, leader changed from <none> to c0b95448a08d4aef93071488929cf25e (127.16.190.194). New cstate: current_term: 1 leader_uuid: "c0b95448a08d4aef93071488929cf25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0b95448a08d4aef93071488929cf25e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 } health_report { overall_health: UNKNOWN } } }
I20260927 21:03:33.155560 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.145868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:35215 (local address 127.16.190.193:33773)
0927 21:03:33.147004 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.147006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.147015 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:03:33.147041 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.147042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.147052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.147100 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:03:33.149006 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.149754 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.153529 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.154092 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.154178 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.154190 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.155418 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.155490 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111}
I20260927 21:03:33.155792 28525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aa9eacfd24f5abd2a7f2d76aadc1d" candidate_uuid: "c0b95448a08d4aef93071488929cf25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" is_pre_election: true
I20260927 21:03:33.155922 28525 raft_consensus.cc:2454] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0b95448a08d4aef93071488929cf25e in term 0.
I20260927 21:03:33.160924 28873 leader_election.cc:288] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [CANDIDATE]: Term 1 election: Requested vote from peers c65b4d97b96d4ceb855da0028f5ad754 (127.16.190.195:34555), 98411c6cc8c547e3bd7e12c9a1aaf6f9 (127.16.190.193:33773)
I20260927 21:03:33.161088 28525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "406aa9eacfd24f5abd2a7f2d76aadc1d" candidate_uuid: "c0b95448a08d4aef93071488929cf25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9"
I20260927 21:03:33.161176 28525 raft_consensus.cc:3037] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:03:33.161972 28525 raft_consensus.cc:2454] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0b95448a08d4aef93071488929cf25e in term 1.
I20260927 21:03:33.173796 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:03:33.170193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.16.190.194:39845)
0927 21:03:33.170232 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:03:33.170234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:03:33.170242 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:03:33.170282 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:03:33.170283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:03:33.170306 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:03:33.170366 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:03:33.170576 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.171338 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.172464 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:03:33.172787 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:03:33.172899 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:03:33.173186 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:03:33.173342 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:03:33.173422 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:03:33.173675 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 21:03:33.173697 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:03:33.176200 28525 raft_consensus.cc:1261] T 406aa9eacfd24f5abd2a7f2d76aadc1d P 98411c6cc8c547e3bd7e12c9a1aaf6f9 [term 1 FOLLOWER]: Refusing update from remote peer c0b95448a08d4aef93071488929cf25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:03:33.176517 28796 raft_consensus.cc:1261] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 1 FOLLOWER]: Refusing update from remote peer c0b95448a08d4aef93071488929cf25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:03:33.177083 28867 consensus_queue.cc:1035] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c65b4d97b96d4ceb855da0028f5ad754" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:03:33.177235 28867 consensus_queue.cc:1035] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98411c6cc8c547e3bd7e12c9a1aaf6f9" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 33773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:03:33.184008 28878 mvcc.cc:205] Tried to move back new op lower bound from 7334064181966069760 to 7334064181878599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334064181966069760}]
I20260927 21:03:33.184206 28881 mvcc.cc:205] Tried to move back new op lower bound from 7334064181966069760 to 7334064181878599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334064181966069760}]
I20260927 21:03:33.186048 28879 mvcc.cc:205] Tried to move back new op lower bound from 7334064181966069760 to 7334064181878599680. Current Snapshot: MvccSnapshot[applied={T|T < 7334064181966069760}]
I20260927 21:03:33.188788 28361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:03:33.188910 28361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:03:33.189662 28361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f5f3df8e22a49d2a875571547ca3090: Sending DeleteTablet for 3 replicas of tablet 406aa9eacfd24f5abd2a7f2d76aadc1d
I20260927 21:03:33.190322 28634 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:03:33 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:39738
I20260927 21:03:33.190582 28508 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:03:33 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:57898
I20260927 21:03:33.190886 28780 tablet_service.cc:1514] Processing DeleteTablet for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:03:33 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:53782
I20260927 21:03:33.191507 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 28443
I20260927 21:03:33.192924 28887 tablet_replica.cc:330] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: stopping tablet replica
I20260927 21:03:33.193044 28887 raft_consensus.cc:2229] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:03:33.193184 28887 raft_consensus.cc:2258] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:03:33.193681 28887 ts_tablet_manager.cc:1905] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:03:33.195703 28889 tablet_replica.cc:330] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: stopping tablet replica
I20260927 21:03:33.195828 28889 raft_consensus.cc:2229] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:03:33.195947 28889 raft_consensus.cc:2258] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:03:33.196168 28887 ts_tablet_manager.cc:1918] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:03:33.196235 28887 log.cc:1196] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/406aa9eacfd24f5abd2a7f2d76aadc1d
I20260927 21:03:33.196478 28887 ts_tablet_manager.cc:1939] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c0b95448a08d4aef93071488929cf25e: Deleting consensus metadata
I20260927 21:03:33.199914 28889 ts_tablet_manager.cc:1905] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:03:33.199970 28347 catalog_manager.cc:4841] TS c0b95448a08d4aef93071488929cf25e (127.16.190.194:39845): tablet 406aa9eacfd24f5abd2a7f2d76aadc1d (table test-table [id=c7881419547247e888e289eca97c6324]) successfully deleted
W20260927 21:03:33.201220 28347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:33773 (error 108)
I20260927 21:03:33.201804 28889 ts_tablet_manager.cc:1918] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:03:33.201887 28889 log.cc:1196] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/406aa9eacfd24f5abd2a7f2d76aadc1d
I20260927 21:03:33.202543 28889 ts_tablet_manager.cc:1939] T 406aa9eacfd24f5abd2a7f2d76aadc1d P c65b4d97b96d4ceb855da0028f5ad754: Deleting consensus metadata
I20260927 21:03:33.203050 28346 catalog_manager.cc:4841] TS c65b4d97b96d4ceb855da0028f5ad754 (127.16.190.195:34555): tablet 406aa9eacfd24f5abd2a7f2d76aadc1d (table test-table [id=c7881419547247e888e289eca97c6324]) successfully deleted
W20260927 21:03:33.207098 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:03:33.201711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:33773 (local address 127.0.0.1:57910)
0927 21:03:33.201762 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:03:33.201775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:03:33.201819 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:03:33.207038 (+ 5219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:33773: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 21:03:33.207356 28347 catalog_manager.cc:4584] TS 98411c6cc8c547e3bd7e12c9a1aaf6f9 (127.16.190.193:33773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 406aa9eacfd24f5abd2a7f2d76aadc1d: Network error: Client connection negotiation failed: client connection to 127.16.190.193:33773: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:03:33.208684 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 28580
I20260927 21:03:33.227118 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 28717
I20260927 21:03:33.232537 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 28330
I20260927 21:03:33.239684 17147 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:03:33.584875 17147 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:03:33.585035 17147 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:03:33.929874 17147 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:03:33.930 UTC [27916] LOG: received smart shutdown request
2026-09-27 21:03:33.934 UTC [27916] LOG: background worker "logical replication launcher" (PID 27925) exited with exit code 1
2026-09-27 21:03:33.934 UTC [27920] LOG: shutting down
2026-09-27 21:03:34.135 UTC [27916] LOG: database system is shut down
2026-09-27T21:03:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48599 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:03:34 dist-test-slave-6c7s krb5kdc[28899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:03:34 dist-test-slave-6c7s krb5kdc[28899](info): set up 2 sockets
Sep 27 21:03:34 dist-test-slave-6c7s krb5kdc[28899](info): commencing operation
krb5kdc: starting...
W20260927 21:03:35.321486 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:03:35 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:03:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangeradmin_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangeradmin_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangerlookup_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangerlookup_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
I20260927 21:03:35.487432 17147 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQTa3tm/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/postgres -l logfile start
2026-09-27 21:03:38.220 UTC [28929] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 21:03:38.220 UTC [28929] LOG: listening on IPv4 address "127.16.190.212", port 58397
2026-09-27 21:03:38.225 UTC [28929] LOG: listening on Unix socket "/tmp/.s.PGSQL.58397"
2026-09-27 21:03:38.259 UTC [28932] LOG: database system was shut down at 2026-09-27 21:03:36 UTC
2026-09-27 21:03:38.264 UTC [28929] LOG: database system is ready to accept connections
I20260927 21:03:39.205397 17147 mini_postgres.cc:96] Postgres bound to 58397
2026-09-27 21:03:39.211 UTC [28940] FATAL: database "slave" does not exist
127.16.190.212:58397 - accepting connections
I20260927 21:03:39.211959 17147 mini_ranger.cc:161] Starting Ranger...
I20260927 21:03:39.227619 17147 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 21:03:39.804347 17147 mini_ranger.cc:87] Created ranger Postgres database
I20260927 21:03:39.804445 17147 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin
2026-09-27 21:03:40,068 [I] DB FLAVOR :POSTGRES
2026-09-27 21:03:40,068 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:03:40,068 [I] Checking connection..
2026-09-27 21:03:40,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 21:03:40,306 [I] Checking connection passed.
2026-09-27 21:03:40,307 [I] --------- Verifying version history table ---------
2026-09-27 21:03:40,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:03:40,549 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:03:40,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:03:40,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 21:03:40,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 21:03:40,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 21:03:41.054 UTC [29004] WARNING: there is no transaction in progress
2026-09-27 21:03:41,061 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 21:03:41,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 21:03:41,319 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 21:03:41,319 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:03:41,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 21:03:41,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 21:03:41,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6c7s.c.gcp-upstream.internal','N') ;"
2026-09-27 21:03:42,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:03:42,344 [I] Table x_portal_user does not exist in database ranger
2026-09-27 21:03:42,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:03:42,594 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 21:03:42,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:03:42,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:03:43,099 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 21:03:43,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 21:03:44.166 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.184 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.202 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.220 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.239 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.259 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.326 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.334 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.350 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.361 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.373 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.389 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.400 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.407 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.415 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:44.423 UTC [29112] WARNING: there is no transaction in progress
2026-09-27 21:03:45,176 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 21:03:45,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 21:03:45,433 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 21:03:45,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 21:03:45,678 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 21:03:45,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 21:03:45,934 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 21:03:45,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 21:03:46,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 21:03:46,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6c7s.c.gcp-upstream.internal';"
2026-09-27 21:03:46,658 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 21:03:46,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 21:03:46,903 [I] DB_PATCHES have already been applied
I20260927 21:03:46.909477 17147 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQTa3tm/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 21:03:46.909549 17147 mini_ranger.cc:193] Using host: 127.16.190.212
I20260927 21:03:46.912652 17147 mini_ranger.cc:227] Ranger admin URL: http://127.16.190.212:36043
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 9:03:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 21:03:48.035427 17147 mini_ranger.cc:160] Time spent starting Ranger: real 8.823s user 0.000s sys 0.009s
Sep 27, 2026 9:04:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 9:04:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 9:04:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 9:04:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 9:04:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 21:04:10.771409 17147 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.190.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.190.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangerkms_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.190.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/rangerkms_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.190.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.190.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.190.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/HTTP_127.16.190.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 21:04:10 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543050, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 21:04:10.882686 17147 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 21:04:10.892952 17147 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 21:04:12.007607 17147 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 21:04:12.007699 17147 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms
I20260927 21:04:12.007719 17147 mini_ranger_kms.cc:227] Using postgres at 127.16.190.212:58397
2026-09-27 21:04:12,184 [I] DB FLAVOR :POSTGRES
2026-09-27 21:04:12,184 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 21:04:12,185 [I] Checking connection
2026-09-27 21:04:12,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 21:04:12,432 [I] connection success
2026-09-27 21:04:12,433 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 21:04:12,433 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 21:04:12,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 21:04:12,702 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 21:04:12,703 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 21:04:12,703 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 21:04:12,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.190.212:58397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 21:04:12,980 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 21:04:13.517531 17147 mini_ranger_kms.cc:310] Created kms service
I20260927 21:04:13.737210 17147 mini_ranger_kms.cc:326] Created kudu user
I20260927 21:04:13.783608 17147 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 21:04:13 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543050, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:14.246091 17147 mini_ranger_kms.cc:384] Added ranger policy
I20260927 21:04:14.246212 17147 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQTa3tm/build/release/bin/postgresql.jar:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQTa3tm/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 21:04:14.246229 17147 mini_ranger_kms.cc:240] Using host: 127.16.190.212
I20260927 21:04:14.249718 17147 mini_ranger_kms.cc:276] Ranger KMS PID: 29293
I20260927 21:04:14.249806 17147 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.190.212:33903
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 9:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 21:04:16 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543056, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:16.527241 17147 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.645s user 0.018s sys 0.141s
I20260927 21:04:16.527333 17147 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 21:04:16.527379 17147 mini_ranger_kms.cc:401] 127.16.190.212:33903/kms/v1/keys
Sep 27 21:04:16 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543056, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
Sep 27 21:04:17 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543056, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.190.212@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
Sep 27 21:04:19 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:04:19.529148 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37585
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37585
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:19.638624 29343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:19.638841 29343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:19.638864 29343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:19.640065 29343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:19.640110 29343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:19.640125 29343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:19.640137 29343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:19.640149 29343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:19.641825 29343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.190.212:33903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37585
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:19.642097 29343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:19.642352 29343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:19.648423 29349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:19.648437 29343 server_base.cc:1034] running on GCE node
W20260927 21:04:19.648442 29348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:19.648494 29351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:19.649034 29343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:19.649392 29343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:19.650553 29343 hybrid_clock.cc:648] HybridClock initialized: now 1790543059650528 us; error 43 us; skew 500 ppm
Sep 27 21:04:19 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543059, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:19.653396 29343 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:19.653970 29343 webserver.cc:460] Webserver started at http://127.16.190.254:45395/ using document root <none> and password file <none>
I20260927 21:04:19.654242 29343 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:19.654297 29343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:19.654434 29343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:04:19 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543059, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:19.706715 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "364a4040bf294ee19b40dd98e4d527a4"
format_stamp: "Formatted at 2026-09-27 21:04:19 on dist-test-slave-6c7s"
server_key: "c00d7663c4e3cd0f62c2ed061681299a"
server_key_iv: "2831859cf7f05a263a1edde6320a74c5"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:19.707166 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "364a4040bf294ee19b40dd98e4d527a4"
format_stamp: "Formatted at 2026-09-27 21:04:19 on dist-test-slave-6c7s"
server_key: "c00d7663c4e3cd0f62c2ed061681299a"
server_key_iv: "2831859cf7f05a263a1edde6320a74c5"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:19.708693 29343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260927 21:04:19.724081 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:19.724646 29343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:19.724766 29343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "364a4040bf294ee19b40dd98e4d527a4"
format_stamp: "Formatted at 2026-09-27 21:04:19 on dist-test-slave-6c7s"
server_key: "c00d7663c4e3cd0f62c2ed061681299a"
server_key_iv: "2831859cf7f05a263a1edde6320a74c5"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:19.724895 29343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:19.745146 29343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:19.747416 29343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:19.747581 29343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:19.752542 29343 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:37585
I20260927 21:04:19.752573 29410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:37585 every 8 connection(s)
I20260927 21:04:19.753021 29343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:19.753743 29411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:19.756060 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Bootstrap starting.
I20260927 21:04:19.756187 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29343
I20260927 21:04:19.756279 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:04:19.756450 17147 external_mini_cluster.cc:1428] Setting key ea275c49eec9e72548e8c72c3cab03b0
I20260927 21:04:19.756868 29411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:19.757296 29411 log.cc:824] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:19.758037 29411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: No bootstrap required, opened a new log
I20260927 21:04:19.759874 29411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:19.760051 29411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:19.760114 29411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364a4040bf294ee19b40dd98e4d527a4, State: Initialized, Role: FOLLOWER
I20260927 21:04:19.760257 29411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:19.760357 29411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:19.760412 29411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:19.760465 29411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:19.761217 29411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:19.761325 29411 leader_election.cc:302] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 364a4040bf294ee19b40dd98e4d527a4; no voters:
I20260927 21:04:19.761523 29411 leader_election.cc:288] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:04:19.761581 29416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:19.761807 29416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 LEADER]: Becoming Leader. State: Replica: 364a4040bf294ee19b40dd98e4d527a4, State: Running, Role: LEADER
I20260927 21:04:19.761859 29411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:19.761957 29416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:19.762331 29416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "364a4040bf294ee19b40dd98e4d527a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } } }
I20260927 21:04:19.762456 29416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:19.762626 29416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364a4040bf294ee19b40dd98e4d527a4. Latest consensus state: current_term: 1 leader_uuid: "364a4040bf294ee19b40dd98e4d527a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } } }
I20260927 21:04:19.762719 29416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:19.762866 29425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:19.763368 29425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:19.763669 29411 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 27 21:04:19 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:19.770330 29425 catalog_manager.cc:1347] Generated new cluster ID: 4064fc5595ea4264a8c3d14be290bab0
I20260927 21:04:19.770542 29425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:04:19.772168 29414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:19.757166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.16.190.254:37585)
0927 21:04:19.757386 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:19.757396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:19.757429 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:04:19.757951 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:19.757960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:19.758028 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:19.758944 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:19.759040 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:19.760520 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:19.761641 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:19.762264 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:19.762355 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:19.766326 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:19.766385 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:19.766386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:19.766504 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:19.769577 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:19.770042 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:19.770045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:19.770046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:19.770142 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:19.770395 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:19.770397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:19.770397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:19.770808 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:19.771416 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:19.771513 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 21:04:19.771872 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260927 21:04:19.805022 29425 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:04:19.805855 29425 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:19.818420 29425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Generated new TSK 0
I20260927 21:04:19.818742 29425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:04:20.565277 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:20.681294 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:20.681522 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:20.681550 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:20.682729 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:20.682766 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:20.682792 29456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:20.684290 29456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:20.684496 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:20.684693 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:20.690026 29461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:20.690016 29464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:20.690011 29462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:20.690738 29456 server_base.cc:1034] running on GCE node
I20260927 21:04:20.691006 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:20.691337 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:20.692538 29456 hybrid_clock.cc:648] HybridClock initialized: now 1790543060692526 us; error 32 us; skew 500 ppm
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:20.695844 29456 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:20.696431 29456 webserver.cc:460] Webserver started at http://127.16.190.193:32795/ using document root <none> and password file <none>
I20260927 21:04:20.696678 29456 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:20.696746 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:20.696933 29456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:20.712417 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "50d7b085e7f94d2e9b51df51abc04593"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "188d278ad53e87b2b3b9ac5ead328547"
server_key_iv: "cb51b8f777b5bfd69e39ef0ad728ee12"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.713121 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "50d7b085e7f94d2e9b51df51abc04593"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "188d278ad53e87b2b3b9ac5ead328547"
server_key_iv: "cb51b8f777b5bfd69e39ef0ad728ee12"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.715826 29456 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 21:04:20.731935 29471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:20.732132 29456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:20.732247 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "50d7b085e7f94d2e9b51df51abc04593"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "188d278ad53e87b2b3b9ac5ead328547"
server_key_iv: "cb51b8f777b5bfd69e39ef0ad728ee12"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.732368 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:20.749410 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:20.752511 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:20.752749 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:20.753065 29456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:20.753515 29456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:20.753616 29456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:20.753724 29456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:20.753803 29456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:20.761471 29456 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:37301
I20260927 21:04:20.762042 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:20.762600 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:37301 every 8 connection(s)
I20260927 21:04:20.766064 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29456
I20260927 21:04:20.766157 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:04:20.766348 17147 external_mini_cluster.cc:1428] Setting key 32a70da0ff14ad98999386748718af6d
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:20.774550 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:20.763377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:54319 (local address 127.16.190.254:37585)
0927 21:04:20.763522 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:20.763524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:20.763816 (+ 292us) server_negotiation.cc:407] Connection header received
0927 21:04:20.764543 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:20.764544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:20.764560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:20.764644 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:20.765486 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:20.766260 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:20.767832 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:20.768159 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:20.768229 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:20.771041 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:20.771043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:20.771043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:20.771052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:20.772974 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:20.773466 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:20.773468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:20.773468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:20.773542 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:20.773828 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:20.773828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:20.773829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:20.773964 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:20.774044 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:20.774347 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 21:04:20.774447 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260927 21:04:20.774745 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:20.763348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.193:54319)
0927 21:04:20.763632 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:20.763688 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:20.764412 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:20.764745 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:20.764750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:20.765062 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:20.765367 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:20.765375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:20.766368 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:20.766369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:20.767730 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:20.767733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:20.768313 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:20.768314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:20.768468 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:20.769167 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:20.769178 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:20.770912 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:20.773089 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:20.773098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:20.773101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:20.773366 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:20.773614 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:20.773615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:20.773615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:20.773752 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:20.774034 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:20.774041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:20.774201 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 21:04:20.774315 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:04:20.774479 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260927 21:04:20.775501 29587 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:20.775632 29587 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:20.775832 29587 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:04:20.777143 29375 ts_manager.cc:194] Registered new tserver with Master: 50d7b085e7f94d2e9b51df51abc04593 (127.16.190.193:37301)
I20260927 21:04:20.778106 29375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:54319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:04:20.801084 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:20.922277 29593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:20.922587 29593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:20.922632 29593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:20.924476 29593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:20.924549 29593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:20.924598 29593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:04:20.926987 29593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:20.927420 29593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:20.927752 29593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:20.934765 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:20.934872 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:20.934760 29601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:20.935551 29593 server_base.cc:1034] running on GCE node
I20260927 21:04:20.935796 29593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:20.936121 29593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:20.937305 29593 hybrid_clock.cc:648] HybridClock initialized: now 1790543060937235 us; error 85 us; skew 500 ppm
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:20.940743 29593 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:20.941393 29593 webserver.cc:460] Webserver started at http://127.16.190.194:36403/ using document root <none> and password file <none>
I20260927 21:04:20.941679 29593 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:20.941753 29593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:20.941939 29593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:04:20 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:20.959326 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "1be5407bf862417299f8719ea6fb7e6a"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "f3637618e08cd186914cb7327161301b"
server_key_iv: "bca86949cac9a100eeca6fcf1aa52545"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.959933 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "1be5407bf862417299f8719ea6fb7e6a"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "f3637618e08cd186914cb7327161301b"
server_key_iv: "bca86949cac9a100eeca6fcf1aa52545"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.965497 29593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260927 21:04:20.993963 29610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:20.994298 29593 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.001s
I20260927 21:04:20.994424 29593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "1be5407bf862417299f8719ea6fb7e6a"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "f3637618e08cd186914cb7327161301b"
server_key_iv: "bca86949cac9a100eeca6fcf1aa52545"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:20.994560 29593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:21.005505 29593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:21.008147 29593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:21.008383 29593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:21.008698 29593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:21.009229 29593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:21.009333 29593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.009405 29593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:21.009464 29593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.015934 29593 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:42973
I20260927 21:04:21.016467 29593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:04:21.019551 29723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:42973 every 8 connection(s)
I20260927 21:04:21.024919 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29593
I20260927 21:04:21.025025 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:04:21.025215 17147 external_mini_cluster.cc:1428] Setting key d9495c32caa6fbacbb669d185b4b1a31
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543060, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:21.032184 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.020522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:42359 (local address 127.16.190.254:37585)
0927 21:04:21.020592 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:21.020595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:21.021180 (+ 585us) server_negotiation.cc:407] Connection header received
0927 21:04:21.021923 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:21.021924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:21.021942 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:21.021977 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:21.022862 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.023690 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.025251 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.025535 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.025585 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.028357 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:21.028359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:21.028359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:21.028367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:21.030365 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.030952 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.030954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.030954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.031017 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.031330 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.031331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.031331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.031481 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:21.031529 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:21.031956 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 21:04:21.032058 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:04:21.033034 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.020207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.194:42359)
0927 21:04:21.021040 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:21.021085 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:21.021779 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:21.022184 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:21.022189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:21.022451 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:21.022745 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.022752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.023809 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.023809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.025136 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.025139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.025692 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.025693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.025833 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.026457 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:21.026468 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:21.028228 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:21.030506 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.030515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.030518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.030836 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.031094 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.031095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.031095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.031244 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.031582 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:21.031588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:21.031771 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 21:04:21.032567 (+ 796us) client_negotiation.cc:240] Negotiation successful
0927 21:04:21.032757 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":105,"threads_started":1}
I20260927 21:04:21.034011 29724 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:21.034111 29724 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.034308 29724 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:21.034715 29375 ts_manager.cc:194] Registered new tserver with Master: 1be5407bf862417299f8719ea6fb7e6a (127.16.190.194:42973)
I20260927 21:04:21.035184 29375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:42359
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:04:21.061940 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:21.183604 29730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:21.184003 29730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:21.184091 29730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:21.186261 29730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:21.186443 29730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:21.186533 29730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:21.189220 29730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:21.189759 29730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:21.190114 29730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:21.197175 29738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.197175 29736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.197491 29735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:21.200639 29730 server_base.cc:1034] running on GCE node
I20260927 21:04:21.200949 29730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:21.201283 29730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:21.202479 29730 hybrid_clock.cc:648] HybridClock initialized: now 1790543061202444 us; error 41 us; skew 500 ppm
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:21.206095 29730 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:21.206691 29730 webserver.cc:460] Webserver started at http://127.16.190.195:46091/ using document root <none> and password file <none>
I20260927 21:04:21.207022 29730 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:21.207096 29730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:21.207264 29730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:21.227473 29730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "8d413f42e906481eb1ddeb3fc6c3d088"
format_stamp: "Formatted at 2026-09-27 21:04:21 on dist-test-slave-6c7s"
server_key: "1b91bd7f778f0a41e7bfd97ba2957c4a"
server_key_iv: "8106d657289059766de36a57512c8751"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.227980 29730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "8d413f42e906481eb1ddeb3fc6c3d088"
format_stamp: "Formatted at 2026-09-27 21:04:21 on dist-test-slave-6c7s"
server_key: "1b91bd7f778f0a41e7bfd97ba2957c4a"
server_key_iv: "8106d657289059766de36a57512c8751"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.230304 29730 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 21:04:21.244652 29745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.244920 29730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:04:21.245028 29730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "8d413f42e906481eb1ddeb3fc6c3d088"
format_stamp: "Formatted at 2026-09-27 21:04:21 on dist-test-slave-6c7s"
server_key: "1b91bd7f778f0a41e7bfd97ba2957c4a"
server_key_iv: "8106d657289059766de36a57512c8751"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.245141 29730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:21.269343 29730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:21.272076 29730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:21.272296 29730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:21.272624 29730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:21.273065 29730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:21.273139 29730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.273190 29730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:21.273236 29730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.280215 29730 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34641
I20260927 21:04:21.280516 29858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34641 every 8 connection(s)
I20260927 21:04:21.280766 29730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:04:21.287096 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29730
I20260927 21:04:21.287232 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:04:21.287420 17147 external_mini_cluster.cc:1428] Setting key 31bb97555da5206bcd95f35188bf5660
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:21.292398 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.281597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:41597 (local address 127.16.190.254:37585)
0927 21:04:21.281676 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:21.281678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:21.281945 (+ 267us) server_negotiation.cc:407] Connection header received
0927 21:04:21.282658 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:21.282659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:21.282672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:21.282748 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:21.283507 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.284289 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.285820 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.286115 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.286162 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.288951 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:21.288953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:21.288953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:21.288962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:21.290745 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.291260 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.291262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.291262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.291321 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.291635 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.291636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.291636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.291779 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:21.291826 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:21.292209 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 21:04:21.292298 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:04:21.293033 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.281570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.195:41597)
0927 21:04:21.281786 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:21.281838 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:21.282527 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:21.282824 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:21.282828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:21.283083 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:21.283373 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.283380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.284409 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.284410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.285713 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.285716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.286253 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.286254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.286399 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.287169 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:21.287179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:21.288800 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:21.290842 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.290850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.290853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.291157 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.291396 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.291397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.291397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.291548 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.291878 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:21.291884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:21.292052 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 21:04:21.292560 (+ 508us) client_negotiation.cc:240] Negotiation successful
0927 21:04:21.292736 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260927 21:04:21.293695 29859 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:21.293808 29859 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.294027 29859 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:21.294458 29375 ts_manager.cc:194] Registered new tserver with Master: 8d413f42e906481eb1ddeb3fc6c3d088 (127.16.190.195:34641)
I20260927 21:04:21.294972 29375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:41597
I20260927 21:04:21.299006 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:04:21.299827 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29456
I20260927 21:04:21.306002 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29593
I20260927 21:04:21.312000 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29730
I20260927 21:04:21.317117 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29343
I20260927 21:04:21.322453 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37585
--webserver_interface=127.16.190.254
--webserver_port=45395
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37585
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:21.405344 29862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:21.405558 29862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:21.405580 29862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:21.406648 29862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:21.406683 29862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:21.406697 29862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:21.406709 29862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:21.406721 29862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:21.408388 29862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37585
--ranger_config_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=45395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:21.408600 29862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:21.408769 29862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:04:21.414144 29862 server_base.cc:1034] running on GCE node
W20260927 21:04:21.414172 29870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.414186 29868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.414214 29867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:21.414906 29862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:21.415235 29862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:21.416451 29862 hybrid_clock.cc:648] HybridClock initialized: now 1790543061416424 us; error 43 us; skew 500 ppm
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:21.419118 29862 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:21.419637 29862 webserver.cc:460] Webserver started at http://127.16.190.254:45395/ using document root <none> and password file <none>
I20260927 21:04:21.419872 29862 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:21.419955 29862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:21.481917 29862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:21.482965 29878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.483170 29862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:21.483268 29862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "364a4040bf294ee19b40dd98e4d527a4"
format_stamp: "Formatted at 2026-09-27 21:04:19 on dist-test-slave-6c7s"
server_key: "c00d7663c4e3cd0f62c2ed061681299a"
server_key_iv: "2831859cf7f05a263a1edde6320a74c5"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.483678 29862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:21.499028 29862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:21.501343 29862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:21.501554 29862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:21.505816 29862 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:37585
I20260927 21:04:21.505987 29930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:37585 every 8 connection(s)
I20260927 21:04:21.506291 29862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:21.507663 29931 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:04:21.508335 29931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Bootstrap starting.
I20260927 21:04:21.509024 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29862
I20260927 21:04:21.509310 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:37301
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:37585
--webserver_port=32795
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:21.516901 29931 log.cc:824] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:21.518790 29931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:04:21.519117 29931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Bootstrap complete.
I20260927 21:04:21.521714 29931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:21.522001 29931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 364a4040bf294ee19b40dd98e4d527a4, State: Initialized, Role: FOLLOWER
I20260927 21:04:21.522205 29931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:21.522287 29931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:21.522341 29931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:21.522396 29931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:04:21.523483 29931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:21.523619 29931 leader_election.cc:302] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 364a4040bf294ee19b40dd98e4d527a4; no voters:
I20260927 21:04:21.523829 29931 leader_election.cc:288] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:04:21.523904 29935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:04:21.524098 29931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:21.524147 29935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [term 2 LEADER]: Becoming Leader. State: Replica: 364a4040bf294ee19b40dd98e4d527a4, State: Running, Role: LEADER
I20260927 21:04:21.524348 29935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } }
I20260927 21:04:21.524636 29936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "364a4040bf294ee19b40dd98e4d527a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } } }
I20260927 21:04:21.524700 29936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:21.524893 29937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 364a4040bf294ee19b40dd98e4d527a4. Latest consensus state: current_term: 2 leader_uuid: "364a4040bf294ee19b40dd98e4d527a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "364a4040bf294ee19b40dd98e4d527a4" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37585 } } }
I20260927 21:04:21.524973 29937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:21.525250 29942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:21.525771 29942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:21.525904 29942 catalog_manager.cc:1259] Loaded cluster ID: 4064fc5595ea4264a8c3d14be290bab0
I20260927 21:04:21.525986 29942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:04:21.526221 29931 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 21:04:21.527376 29942 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:21.527568 29942 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Loaded TSK: 0
I20260927 21:04:21.527781 29942 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:04:21.625938 29933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:21.626238 29933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:21.626274 29933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:21.628248 29933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:21.628319 29933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:21.628364 29933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:21.630825 29933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:37301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=32795
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:21.631112 29933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:21.631439 29933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:21.637791 29969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.637811 29971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.638402 29968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:21.638712 29933 server_base.cc:1034] running on GCE node
I20260927 21:04:21.639084 29933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:21.639396 29933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:21.640578 29933 hybrid_clock.cc:648] HybridClock initialized: now 1790543061640546 us; error 47 us; skew 500 ppm
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:21.643718 29933 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:21.644235 29933 webserver.cc:460] Webserver started at http://127.16.190.193:32795/ using document root <none> and password file <none>
I20260927 21:04:21.644430 29933 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:21.644513 29933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:21.670077 29933 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260927 21:04:21.671147 29979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.671429 29933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:21.671538 29933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "50d7b085e7f94d2e9b51df51abc04593"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "188d278ad53e87b2b3b9ac5ead328547"
server_key_iv: "cb51b8f777b5bfd69e39ef0ad728ee12"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.672068 29933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:21.690100 29933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:21.692526 29933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:21.692701 29933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:21.693027 29933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:21.693418 29933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:21.693476 29933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.693511 29933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:21.693555 29933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.701387 29933 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:37301
I20260927 21:04:21.701809 29933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:21.701911 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:37301 every 8 connection(s)
I20260927 21:04:21.705462 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 29933
I20260927 21:04:21.705794 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:42973
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:37585
--webserver_port=36403
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:21.717458 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.702803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:41757 (local address 127.16.190.254:37585)
0927 21:04:21.703013 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:21.703024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:21.703127 (+ 103us) server_negotiation.cc:407] Connection header received
0927 21:04:21.704012 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:21.704020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:21.704090 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:21.704256 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:21.705018 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.706601 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.708155 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.708948 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.709141 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.712291 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:21.712298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:21.712298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:21.712317 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:21.714963 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.715636 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.715640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.715641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.715770 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.716144 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.716146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.716146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.716492 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:21.716591 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:21.717071 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 21:04:21.717215 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260927 21:04:21.717533 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.702726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.193:41757)
0927 21:04:21.702958 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:21.703012 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:21.703668 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:21.704322 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:21.704326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:21.704563 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:21.704867 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.704875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.706736 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.706737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.708040 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.708043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.709142 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.709144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.709295 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.710034 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:21.710046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:21.712133 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:21.715111 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.715120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.715123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.715516 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.715869 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.715870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.715870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.716052 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.716590 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:21.716598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:21.716802 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 21:04:21.716988 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 21:04:21.717198 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260927 21:04:21.720033 30093 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:21.720148 30093 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.720373 30093 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:21.721347 30093 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:04:21.721419 30093 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.721481 30093 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:21.721819 30093 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.721905 30093 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:21.722222 30093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:04:21.825760 30097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:21.826150 30097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:21.826238 30097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:21.828075 30097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:21.828192 30097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:21.828281 30097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:04:21.830772 30097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:42973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=36403
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:21.831063 30097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:21.831422 30097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:21.837641 30103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:21.837877 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:21.838367 30097 server_base.cc:1034] running on GCE node
W20260927 21:04:21.839558 30102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:21.839839 30097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:21.840145 30097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:21.841391 30097 hybrid_clock.cc:648] HybridClock initialized: now 1790543061841364 us; error 45 us; skew 500 ppm
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:21.844640 30097 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:21.845232 30097 webserver.cc:460] Webserver started at http://127.16.190.194:36403/ using document root <none> and password file <none>
I20260927 21:04:21.845609 30097 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:21.845753 30097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:21.863286 30097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:21.864378 30113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.864612 30097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:21.864702 30097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "1be5407bf862417299f8719ea6fb7e6a"
format_stamp: "Formatted at 2026-09-27 21:04:20 on dist-test-slave-6c7s"
server_key: "f3637618e08cd186914cb7327161301b"
server_key_iv: "bca86949cac9a100eeca6fcf1aa52545"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:21.865162 30097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:21.901419 30097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:21.903961 30097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:21.904177 30097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:21.904491 30097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:21.904973 30097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:21.905025 30097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.905056 30097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:21.905074 30097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:21.912926 30097 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:42973
I20260927 21:04:21.913014 30226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:42973 every 8 connection(s)
I20260927 21:04:21.913523 30097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:04:21 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543061, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:21.922933 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.914629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:57037 (local address 127.16.190.254:37585)
0927 21:04:21.914675 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:21.914677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:21.914689 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:04:21.915127 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:21.915129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:21.915147 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:21.915228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:21.915819 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.916325 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.917471 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.917684 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.917770 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.919969 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:21.919970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:21.919971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:21.919979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:21.921459 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.921903 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.921904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.921904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.921961 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:21.922223 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:21.922224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:21.922224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:21.922343 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:21.922420 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:21.922708 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:04:21.922795 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:04:21.922967 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30097
I20260927 21:04:21.923233 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.190.212:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:34641
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:37585
--webserver_port=46091
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:21.925271 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:21.914322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.194:57037)
0927 21:04:21.914511 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:21.914548 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:21.915003 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:21.915287 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:21.915290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:21.915484 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:21.915715 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.915722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.916456 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.916457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.917363 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:21.917366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:21.917797 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:21.917799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:21.917923 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:21.918486 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:21.918494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:21.919872 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:21.921562 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.921569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.921571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.921804 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.922029 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:21.922030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:21.922031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:21.922154 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:21.922412 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:21.922418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:21.922568 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 21:04:21.922674 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:04:21.924983 (+ 2309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260927 21:04:21.926164 30227 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:21.926314 30227 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.926568 30227 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:21.927362 30227 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:04:21.927481 30227 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.927587 30227 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:21.927966 30227 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:21.928056 30227 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:21.928390 30227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:04:22.039374 30230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:22.039727 30230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:22.039795 30230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:22.041742 30230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:22.041862 30230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:22.041929 30230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:22.044234 30230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.190.212:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:34641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=46091
--tserver_master_addrs=127.16.190.254:37585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:22.044654 30230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:22.044976 30230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:22.052955 30239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:22.052971 30237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:22.053081 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:22.053660 30230 server_base.cc:1034] running on GCE node
I20260927 21:04:22.053879 30230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:22.054157 30230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:22.055362 30230 hybrid_clock.cc:648] HybridClock initialized: now 1790543062055334 us; error 43 us; skew 500 ppm
Sep 27 21:04:22 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:22.058804 30230 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:22.059492 30230 webserver.cc:460] Webserver started at http://127.16.190.195:46091/ using document root <none> and password file <none>
I20260927 21:04:22.059798 30230 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:22.059880 30230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 21:04:22 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:22.083930 30230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:22.085289 30247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:22.085543 30230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:22.085633 30230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "8d413f42e906481eb1ddeb3fc6c3d088"
format_stamp: "Formatted at 2026-09-27 21:04:21 on dist-test-slave-6c7s"
server_key: "1b91bd7f778f0a41e7bfd97ba2957c4a"
server_key_iv: "8106d657289059766de36a57512c8751"
server_key_version: "kuduclusterkey@0"
I20260927 21:04:22.086127 30230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:22.101879 30230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:22.104413 30230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:22.104625 30230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:22.104946 30230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:22.105388 30230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:22.105468 30230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:22.105526 30230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:22.105571 30230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:22.112210 30230 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34641
I20260927 21:04:22.112762 30230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:04:22.115931 30360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34641 every 8 connection(s)
I20260927 21:04:22.121645 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30230
Sep 27 21:04:22 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:22.127642 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:22.116646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:51661 (local address 127.16.190.254:37585)
0927 21:04:22.116697 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:22.116699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:22.117031 (+ 332us) server_negotiation.cc:407] Connection header received
0927 21:04:22.117744 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:22.117746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:22.117760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:22.117830 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:22.118503 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.119251 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.120276 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.120587 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.120669 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:22.123910 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:22.123911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:22.123911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:22.123918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:22.125732 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:22.126236 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:22.126238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:22.126238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:22.126304 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:22.126864 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:22.126866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:22.126866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:22.126999 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:22.127046 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:22.127458 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 21:04:22.127547 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:04:22.127869 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:22.116637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37585 (local address 127.16.190.195:51661)
0927 21:04:22.116859 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:22.116915 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:22.117609 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:22.117923 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:22.117928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:22.118191 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:22.118403 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:22.118408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.119353 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.119355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:22.120180 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:22.120182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.120726 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.120728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:22.120905 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:22.121576 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:22.121587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:22.123791 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:22.125846 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:22.125853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:22.125856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:22.126148 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:22.126630 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:22.126631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:22.126631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:22.126750 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:22.127098 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:22.127105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:22.127300 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:04:22.127435 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 21:04:22.127602 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260927 21:04:22.128473 30361 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37585
I20260927 21:04:22.128604 30361 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:22.128839 30361 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:22.129525 30361 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:37585 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 21:04:22.129601 30361 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:22.129665 30361 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:22.130044 30361 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:22.130126 30361 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
W20260927 21:04:22.130447 30361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:04:22.130910 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:22.122101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.16.190.254:37585)
0927 21:04:22.122434 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:22.122435 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:22.122440 (+ 5us) server_negotiation.cc:407] Connection header received
0927 21:04:22.122457 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:22.122458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:22.122494 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:22.122553 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:22.123327 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.124572 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.125984 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:22.126292 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:22.126375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:22.127902 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:22.127904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:22.127904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:22.127910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:22.129631 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:22.130051 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:22.130052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:22.130053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:22.130113 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:22.130305 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:22.130305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:22.130306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:22.130447 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:22.130608 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:22.130723 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 21:04:22.130812 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":66,"threads_started":1}
Sep 27 21:04:22 dist-test-slave-6c7s krb5kdc[28899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:22.722394 30093 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:22.722558 30093 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:22.722949 29895 ts_manager.cc:194] Registered new tserver with Master: 50d7b085e7f94d2e9b51df51abc04593 (127.16.190.193:37301)
I20260927 21:04:22.724001 29895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:41757
I20260927 21:04:22.928633 30227 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:22.928802 30227 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:22.929369 29895 ts_manager.cc:194] Registered new tserver with Master: 1be5407bf862417299f8719ea6fb7e6a (127.16.190.194:42973)
I20260927 21:04:22.929906 29895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:57037
Sep 27 21:04:23 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:23.130805 30361 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:23.130970 30361 heartbeater.cc:511] Master 127.16.190.254:37585 requested a full tablet report, sending...
I20260927 21:04:23.131389 29895 ts_manager.cc:194] Registered new tserver with Master: 8d413f42e906481eb1ddeb3fc6c3d088 (127.16.190.195:34641)
I20260927 21:04:23.131945 29895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:51661
I20260927 21:04:23.133545 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:04:23.142649 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.134521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.16.190.254:37585)
0927 21:04:23.134700 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.134703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.134865 (+ 162us) server_negotiation.cc:407] Connection header received
0927 21:04:23.134946 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.134948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.134960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.135050 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:23.135685 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.136425 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.137526 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.137827 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.137871 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.139432 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:23.139433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:23.139433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:23.139441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:23.141319 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:23.141754 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:23.141755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:23.141755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:23.141815 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:23.142019 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:23.142020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:23.142020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:23.142150 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:23.142222 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.142429 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.142520 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
I20260927 21:04:23.145056 29895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:04:23.146816 29895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:04:23.160983 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34641 (local address 127.0.0.1:39844)
0927 21:04:23.156107 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.156113 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:23.156149 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:23.156453 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:23.156457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:23.156459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:23.156534 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.156536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.158141 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.158142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.159390 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.159393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.160663 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.160664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.160744 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.160762 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:04:23.160830 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:04:23.160888 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":59,"threads_started":1}
I20260927 21:04:23.161695 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.16.190.195:34641)
0927 21:04:23.155745 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.155751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.156203 (+ 452us) server_negotiation.cc:407] Connection header received
0927 21:04:23.156248 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.156250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.156267 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.156353 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:23.156616 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.157747 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.159495 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.160561 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.161476 (+ 915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.161545 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.161572 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.161615 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260927 21:04:23.162655 30277 tablet_service.cc:1467] Processing CreateTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 (DEFAULT_TABLE table=test-table [id=d4814dcca42644028210fc32f4aa483e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:23.163050 30277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c3a03e41c42738508e2dca6ce2c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:23.163851 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:37301 (local address 127.0.0.1:33064)
0927 21:04:23.155779 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.155821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:23.155892 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:23.156468 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:23.156469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:23.156471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:23.156539 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.156541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.159371 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.159372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.160429 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.160431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.163596 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.163597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.163667 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.163672 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:23.163730 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:04:23.163777 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":89,"threads_started":1}
I20260927 21:04:23.163976 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:42973 (local address 127.0.0.1:49518)
0927 21:04:23.156002 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.156010 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:23.156043 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:23.158050 (+ 2007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:23.158051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:23.158053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:23.158097 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.158099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.160898 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.160899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.162052 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.162055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.163717 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.163719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.163785 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.163789 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:04:23.163820 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 21:04:23.163859 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":81,"threads_started":1}
I20260927 21:04:23.164134 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.16.190.194:42973)
0927 21:04:23.156668 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.156672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.156681 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:23.156712 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.156714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.156729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.158175 (+ 1446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:23.158256 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.159258 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.162164 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.162890 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.163908 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.163976 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.164014 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.164059 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":53,"threads_started":1}
I20260927 21:04:23.164250 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.155235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.16.190.193:37301)
0927 21:04:23.156248 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.156252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.156261 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:23.156295 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.156297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.156312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.156377 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:23.156865 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.157920 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.160541 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.163488 (+ 2947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.163596 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.163651 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.164123 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.164173 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":55,"threads_started":1}
I20260927 21:04:23.164855 30161 tablet_service.cc:1467] Processing CreateTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 (DEFAULT_TABLE table=test-table [id=d4814dcca42644028210fc32f4aa483e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:23.165217 30161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c3a03e41c42738508e2dca6ce2c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:23.165412 30020 tablet_service.cc:1467] Processing CreateTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 (DEFAULT_TABLE table=test-table [id=d4814dcca42644028210fc32f4aa483e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:23.165716 30020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1c3a03e41c42738508e2dca6ce2c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:23.169023 30395 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Bootstrap starting.
I20260927 21:04:23.169639 30395 tablet_bootstrap.cc:654] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:23.170061 30395 log.cc:824] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:23.170420 30396 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Bootstrap starting.
I20260927 21:04:23.170807 30395 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: No bootstrap required, opened a new log
I20260927 21:04:23.170903 30395 ts_tablet_manager.cc:1397] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:04:23.173012 30394 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Bootstrap starting.
I20260927 21:04:23.172962 30395 raft_consensus.cc:348] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.173228 30395 raft_consensus.cc:374] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:23.173308 30395 raft_consensus.cc:729] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be5407bf862417299f8719ea6fb7e6a, State: Initialized, Role: FOLLOWER
I20260927 21:04:23.173518 30395 consensus_queue.cc:260] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.173787 30394 tablet_bootstrap.cc:654] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:23.173907 30395 ts_tablet_manager.cc:1428] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:04:23.174078 30227 heartbeater.cc:503] Master 127.16.190.254:37585 was elected leader, sending a full tablet report...
I20260927 21:04:23.174140 30394 log.cc:824] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:23.175163 30396 tablet_bootstrap.cc:654] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:23.175457 30396 log.cc:824] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:23.176333 30396 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: No bootstrap required, opened a new log
I20260927 21:04:23.176420 30396 ts_tablet_manager.cc:1397] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 21:04:23.179735 30396 raft_consensus.cc:348] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.180063 30396 raft_consensus.cc:374] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:23.180145 30396 raft_consensus.cc:729] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d7b085e7f94d2e9b51df51abc04593, State: Initialized, Role: FOLLOWER
I20260927 21:04:23.180334 30396 consensus_queue.cc:260] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.180691 30396 ts_tablet_manager.cc:1428] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 21:04:23.180912 30093 heartbeater.cc:503] Master 127.16.190.254:37585 was elected leader, sending a full tablet report...
I20260927 21:04:23.181921 30394 tablet_bootstrap.cc:492] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: No bootstrap required, opened a new log
I20260927 21:04:23.182053 30394 ts_tablet_manager.cc:1397] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 21:04:23.184891 30394 raft_consensus.cc:348] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.185173 30394 raft_consensus.cc:374] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:23.185247 30394 raft_consensus.cc:729] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d413f42e906481eb1ddeb3fc6c3d088, State: Initialized, Role: FOLLOWER
I20260927 21:04:23.185421 30394 consensus_queue.cc:260] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.185741 30394 ts_tablet_manager.cc:1428] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:04:23.185774 30361 heartbeater.cc:503] Master 127.16.190.254:37585 was elected leader, sending a full tablet report...
W20260927 21:04:23.203255 30094 tablet.cc:2369] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:23.273501 30401 raft_consensus.cc:482] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:23.273788 30401 raft_consensus.cc:504] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.274224 30401 leader_election.cc:288] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1be5407bf862417299f8719ea6fb7e6a (127.16.190.194:42973), 8d413f42e906481eb1ddeb3fc6c3d088 (127.16.190.195:34641)
I20260927 21:04:23.278940 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.274908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:42911 (local address 127.16.190.194:42973)
0927 21:04:23.274957 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.274959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.275117 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:04:23.275221 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.275222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.275232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.275260 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:23.275495 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.276232 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.277907 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.278508 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.278564 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.278576 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.278811 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.278861 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:04:23.279232 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.274403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:42973 (local address 127.16.190.193:42911)
0927 21:04:23.275048 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.275060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:23.275145 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:23.275328 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:23.275328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:23.275330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:23.275408 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.275410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.276340 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.276341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.277773 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.277776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.278659 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.278660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.278730 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.278733 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:04:23.279082 (+ 349us) client_negotiation.cc:240] Negotiation successful
0927 21:04:23.279132 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":118,"threads_started":1}
I20260927 21:04:23.279584 30393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.274824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:33663 (local address 127.16.190.195:34641)
0927 21:04:23.274869 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.274872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.275636 (+ 764us) server_negotiation.cc:407] Connection header received
0927 21:04:23.275743 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.275744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.275753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.275783 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:23.276009 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.276704 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.278372 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.278931 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.278988 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.278996 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.279467 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.279515 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:04:23.279783 30181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c3a03e41c42738508e2dca6ce2c25" candidate_uuid: "50d7b085e7f94d2e9b51df51abc04593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be5407bf862417299f8719ea6fb7e6a" is_pre_election: true
I20260927 21:04:23.279902 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.274628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34641 (local address 127.16.190.193:33663)
0927 21:04:23.275554 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.275567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:23.275666 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:23.275853 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:23.275854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:23.275855 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:23.275926 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.275928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.277038 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.277039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.278254 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:23.278257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.279281 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.279282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:23.279371 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.279375 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:04:23.279780 (+ 405us) client_negotiation.cc:240] Negotiation successful
0927 21:04:23.279825 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":61,"threads_started":1}
I20260927 21:04:23.280006 30181 raft_consensus.cc:2454] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d7b085e7f94d2e9b51df51abc04593 in term 0.
I20260927 21:04:23.280328 29980 leader_election.cc:302] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be5407bf862417299f8719ea6fb7e6a, 50d7b085e7f94d2e9b51df51abc04593; no voters:
I20260927 21:04:23.280306 30315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c3a03e41c42738508e2dca6ce2c25" candidate_uuid: "50d7b085e7f94d2e9b51df51abc04593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" is_pre_election: true
I20260927 21:04:23.280471 30315 raft_consensus.cc:2454] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d7b085e7f94d2e9b51df51abc04593 in term 0.
I20260927 21:04:23.280483 30401 raft_consensus.cc:2781] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:04:23.280527 30401 raft_consensus.cc:482] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:04:23.280587 30401 raft_consensus.cc:3037] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:23.281574 30401 raft_consensus.cc:504] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.281872 30181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c3a03e41c42738508e2dca6ce2c25" candidate_uuid: "50d7b085e7f94d2e9b51df51abc04593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be5407bf862417299f8719ea6fb7e6a"
I20260927 21:04:23.281920 30315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1c3a03e41c42738508e2dca6ce2c25" candidate_uuid: "50d7b085e7f94d2e9b51df51abc04593" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d413f42e906481eb1ddeb3fc6c3d088"
I20260927 21:04:23.281956 30181 raft_consensus.cc:3037] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:23.282001 30315 raft_consensus.cc:3037] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:23.282828 30181 raft_consensus.cc:2454] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d7b085e7f94d2e9b51df51abc04593 in term 1.
I20260927 21:04:23.282963 30315 raft_consensus.cc:2454] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d7b085e7f94d2e9b51df51abc04593 in term 1.
I20260927 21:04:23.283087 29980 leader_election.cc:302] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be5407bf862417299f8719ea6fb7e6a, 50d7b085e7f94d2e9b51df51abc04593; no voters:
I20260927 21:04:23.283115 30401 leader_election.cc:288] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [CANDIDATE]: Term 1 election: Requested vote from peers 1be5407bf862417299f8719ea6fb7e6a (127.16.190.194:42973), 8d413f42e906481eb1ddeb3fc6c3d088 (127.16.190.195:34641)
I20260927 21:04:23.283172 30401 raft_consensus.cc:2781] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:23.283208 30401 raft_consensus.cc:686] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [term 1 LEADER]: Becoming Leader. State: Replica: 50d7b085e7f94d2e9b51df51abc04593, State: Running, Role: LEADER
I20260927 21:04:23.283293 30401 consensus_queue.cc:237] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } }
I20260927 21:04:23.284379 29895 catalog_manager.cc:5495] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d7b085e7f94d2e9b51df51abc04593 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "50d7b085e7f94d2e9b51df51abc04593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50d7b085e7f94d2e9b51df51abc04593" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 } health_report { overall_health: UNKNOWN } } }
I20260927 21:04:23.305907 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.302399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.16.190.193:37301)
0927 21:04:23.302446 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:23.302449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:23.302459 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:04:23.302559 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:23.302560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:23.302583 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:23.302660 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:04:23.302858 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.303635 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.304845 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:23.305158 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:23.305273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:23.305496 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:04:23.305631 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:04:23.305720 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:23.305779 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 21:04:23.305804 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:04:23.308197 30315 raft_consensus.cc:1261] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 1 FOLLOWER]: Refusing update from remote peer 50d7b085e7f94d2e9b51df51abc04593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:23.308635 30401 consensus_queue.cc:1035] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d413f42e906481eb1ddeb3fc6c3d088" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:23.310935 30181 raft_consensus.cc:1261] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 1 FOLLOWER]: Refusing update from remote peer 50d7b085e7f94d2e9b51df51abc04593: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:23.311412 30401 consensus_queue.cc:1035] T df1c3a03e41c42738508e2dca6ce2c25 P 50d7b085e7f94d2e9b51df51abc04593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be5407bf862417299f8719ea6fb7e6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:23.313499 30409 mvcc.cc:205] Tried to move back new op lower bound from 7334064387307249664 to 7334064387209977856. Current Snapshot: MvccSnapshot[applied={T|T < 7334064387307249664}]
I20260927 21:04:23.314265 30410 mvcc.cc:205] Tried to move back new op lower bound from 7334064387307249664 to 7334064387209977856. Current Snapshot: MvccSnapshot[applied={T|T < 7334064387307249664}]
I20260927 21:04:23.316069 29894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:23.316175 29894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:23.322562 29894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 364a4040bf294ee19b40dd98e4d527a4: Sending DeleteTablet for 3 replicas of tablet df1c3a03e41c42738508e2dca6ce2c25
I20260927 21:04:23.323218 30161 tablet_service.cc:1514] Processing DeleteTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:23 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:49518
I20260927 21:04:23.323915 30020 tablet_service.cc:1514] Processing DeleteTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:23 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:33064
I20260927 21:04:23.324731 30277 tablet_service.cc:1514] Processing DeleteTablet for tablet df1c3a03e41c42738508e2dca6ce2c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:23 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:39844
I20260927 21:04:23.325205 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29933
I20260927 21:04:23.327250 30413 mvcc.cc:205] Tried to move back new op lower bound from 7334064387307249664 to 7334064387209977856. Current Snapshot: MvccSnapshot[applied={T|T < 7334064387307249664 or (T in {7334064387307249664})}]
I20260927 21:04:23.327533 30418 tablet_replica.cc:330] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: stopping tablet replica
I20260927 21:04:23.327646 30418 raft_consensus.cc:2229] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:23.327762 30418 raft_consensus.cc:2258] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:23.327942 30420 tablet_replica.cc:330] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: stopping tablet replica
I20260927 21:04:23.328061 30420 raft_consensus.cc:2229] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:23.328176 30420 raft_consensus.cc:2258] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:23.328686 30420 ts_tablet_manager.cc:1905] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 21:04:23 dist-test-slave-6c7s krb5kdc[28899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543062, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for HTTP/127.16.190.212@KRBTEST.COM
I20260927 21:04:23.329185 30418 ts_tablet_manager.cc:1905] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:23.330803 30418 ts_tablet_manager.cc:1918] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:23.330947 30418 log.cc:1196] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/df1c3a03e41c42738508e2dca6ce2c25
I20260927 21:04:23.331286 30418 ts_tablet_manager.cc:1939] T df1c3a03e41c42738508e2dca6ce2c25 P 1be5407bf862417299f8719ea6fb7e6a: Deleting consensus metadata
I20260927 21:04:23.331784 29879 catalog_manager.cc:4841] TS 1be5407bf862417299f8719ea6fb7e6a (127.16.190.194:42973): tablet df1c3a03e41c42738508e2dca6ce2c25 (table test-table [id=d4814dcca42644028210fc32f4aa483e]) successfully deleted
I20260927 21:04:23.334545 30420 ts_tablet_manager.cc:1918] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:23.334640 30420 log.cc:1196] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/df1c3a03e41c42738508e2dca6ce2c25
I20260927 21:04:23.334935 30420 ts_tablet_manager.cc:1939] T df1c3a03e41c42738508e2dca6ce2c25 P 8d413f42e906481eb1ddeb3fc6c3d088: Deleting consensus metadata
I20260927 21:04:23.335440 29879 catalog_manager.cc:4841] TS 8d413f42e906481eb1ddeb3fc6c3d088 (127.16.190.195:34641): tablet df1c3a03e41c42738508e2dca6ce2c25 (table test-table [id=d4814dcca42644028210fc32f4aa483e]) successfully deleted
W20260927 21:04:23.335775 29880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:37301 (error 108)
I20260927 21:04:23.336499 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:23.336382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:37301 (local address 127.0.0.1:33068)
0927 21:04:23.336432 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:23.336463 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:37301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:04:23.336704 29880 catalog_manager.cc:4584] TS 50d7b085e7f94d2e9b51df51abc04593 (127.16.190.193:37301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df1c3a03e41c42738508e2dca6ce2c25: Network error: Client connection negotiation failed: client connection to 127.16.190.193:37301: connect: Connection refused (error 111)
I20260927 21:04:23.342340 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30097
I20260927 21:04:23.348574 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30230
I20260927 21:04:23.356544 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 29862
I20260927 21:04:23.366895 17147 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 21:04:23.712013 17147 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 21:04:23.712147 17147 mini_ranger.cc:66] Stopping Ranger...
I20260927 21:04:24.057492 17147 mini_ranger.cc:68] Stopped Ranger
2026-09-27 21:04:24.057 UTC [28929] LOG: received smart shutdown request
2026-09-27 21:04:24.062 UTC [28929] LOG: background worker "logical replication launcher" (PID 28938) exited with exit code 1
2026-09-27 21:04:24.062 UTC [28933] LOG: shutting down
2026-09-27 21:04:24.283 UTC [28929] LOG: database system is shut down
2026-09-27T21:04:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50147 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:04:24 dist-test-slave-6c7s krb5kdc[30431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:04:24 dist-test-slave-6c7s krb5kdc[30431](info): set up 2 sockets
Sep 27 21:04:24 dist-test-slave-6c7s krb5kdc[30431](info): commencing operation
krb5kdc: starting...
W20260927 21:04:25.445648 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:04:25.571983 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45121
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:25.655289 30444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:25.655548 30444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:25.655586 30444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:25.656793 30444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:25.656876 30444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:25.656903 30444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:25.656934 30444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:25.656961 30444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:25.658478 30444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:25.658746 30444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:25.658970 30444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:25.663774 30450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:25.663810 30452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:25.663776 30449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:25.664039 30444 server_base.cc:1034] running on GCE node
I20260927 21:04:25.664342 30444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:25.664629 30444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:25.665823 30444 hybrid_clock.cc:648] HybridClock initialized: now 1790543065665798 us; error 54 us; skew 500 ppm
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:25.668370 30444 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:25.668931 30444 webserver.cc:460] Webserver started at http://127.16.190.254:46625/ using document root <none> and password file <none>
I20260927 21:04:25.669190 30444 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:25.669257 30444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:25.669407 30444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:25.670475 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "1d6b8e86d247470688b86b24997e84ec"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "c0f5f553aecd7fa257e39c74c0b784ea"
server_key_iv: "b7fd179eb21ef687b67460efa62eb4a1"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.670848 30444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "1d6b8e86d247470688b86b24997e84ec"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "c0f5f553aecd7fa257e39c74c0b784ea"
server_key_iv: "b7fd179eb21ef687b67460efa62eb4a1"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.672183 30444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:25.672976 30459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:25.673158 30444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:25.673247 30444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "1d6b8e86d247470688b86b24997e84ec"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "c0f5f553aecd7fa257e39c74c0b784ea"
server_key_iv: "b7fd179eb21ef687b67460efa62eb4a1"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.673336 30444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:25.684675 30444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:25.686544 30444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:25.686705 30444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:25.690869 30444 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:45121
I20260927 21:04:25.690901 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:45121 every 8 connection(s)
I20260927 21:04:25.691257 30444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:25.692116 30512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:25.693838 30512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:04:25.697280 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30444
I20260927 21:04:25.697397 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:04:25.697582 17147 external_mini_cluster.cc:1428] Setting key eadfdf7984e755887dc9b65eea9daec0
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:25.704155 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.694517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45121 (local address 127.0.0.1:53192)
0927 21:04:25.694679 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.694716 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:25.695153 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:25.695406 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:25.695408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:25.695697 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:25.695847 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.695851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.696904 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.696905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.697987 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.697990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.698460 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.698461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.698552 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.699144 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:25.699153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:25.700530 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:25.702742 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.702747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.702749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.702936 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.703201 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.703201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.703201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.703282 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.703726 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:25.703731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:25.703786 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 21:04:25.703861 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:04:25.703993 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260927 21:04:25.704226 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.694553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.16.190.254:45121)
0927 21:04:25.694709 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:25.694717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:25.694811 (+ 94us) server_negotiation.cc:407] Connection header received
0927 21:04:25.695280 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:25.695282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:25.695298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:25.695383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:25.695944 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.696772 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.698065 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.698374 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.698460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.700620 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:25.700628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:25.700628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:25.700635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:25.702658 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.703017 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.703021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.703021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.703130 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.703350 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.703351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.703352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.703650 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:25.703743 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:25.703920 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 21:04:25.704075 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.16.190.253@KRBTEST.COM; defaulting to no policy
I20260927 21:04:25.705708 30512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:04:25.706089 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.705984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54930)
0927 21:04:25.706029 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.706069 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:04:25.706259 30462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
I20260927 21:04:25.706681 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.706599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54936)
0927 21:04:25.706643 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.706666 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:04:25.706805 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:39969: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
I20260927 21:04:25.725139 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:39969
--webserver_interface=127.16.190.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:25.756026 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } attempt: 1
I20260927 21:04:25.756572 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.756423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54952)
0927 21:04:25.756502 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.756541 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 21:04:25.757005 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.756908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54958)
0927 21:04:25.756952 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.756981 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:04:25.757102 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:39969: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
W20260927 21:04:25.810788 30519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:25.811069 30519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:25.811110 30519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:25.812346 30519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:25.812424 30519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:25.812448 30519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:25.812482 30519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:25.812531 30519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:25.814179 30519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:39969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.190.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:25.814486 30519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:25.814728 30519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:25.820043 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:25.820086 30519 server_base.cc:1034] running on GCE node
W20260927 21:04:25.820068 30525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:25.820067 30524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:25.820619 30519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:25.821041 30519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:25.822230 30519 hybrid_clock.cc:648] HybridClock initialized: now 1790543065822216 us; error 32 us; skew 500 ppm
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:25.825186 30519 init.cc:377] Logged in from keytab as kudu/127.16.190.253@KRBTEST.COM (short username kudu)
I20260927 21:04:25.825268 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } attempt: 2
I20260927 21:04:25.825758 30519 webserver.cc:460] Webserver started at http://127.16.190.253:37379/ using document root <none> and password file <none>
I20260927 21:04:25.825778 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.825620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54966)
0927 21:04:25.825684 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.825722 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:04:25.826026 30519 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:25.826105 30519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:25.826152 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.826045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54982)
0927 21:04:25.826096 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.826123 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:04:25.826243 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:39969: Network error: Client connection negotiation failed: client connection to 127.16.190.253:39969: connect: Connection refused (error 111)
I20260927 21:04:25.826274 30519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:25.827389 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data/instance:
uuid: "d0b0e7223bf44c6e935e6860163fa2a3"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "05f82eec0d178a1aae85a6b4b6df179a"
server_key_iv: "a5d3e9ab0a2ba1a74c9d426be74a25dd"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.827701 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal/instance:
uuid: "d0b0e7223bf44c6e935e6860163fa2a3"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "05f82eec0d178a1aae85a6b4b6df179a"
server_key_iv: "a5d3e9ab0a2ba1a74c9d426be74a25dd"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.829056 30519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:04:25.829944 30534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:25.830111 30519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:25.830207 30519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "d0b0e7223bf44c6e935e6860163fa2a3"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "05f82eec0d178a1aae85a6b4b6df179a"
server_key_iv: "a5d3e9ab0a2ba1a74c9d426be74a25dd"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.830266 30519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:25.842278 30519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:25.845059 30519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:25.845184 30519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:25.849174 30519 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.253:39969
I20260927 21:04:25.849294 30586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.253:39969 every 8 connection(s)
I20260927 21:04:25.849647 30519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
I20260927 21:04:25.850122 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30519
I20260927 21:04:25.850206 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-1/wal/instance
I20260927 21:04:25.850421 17147 external_mini_cluster.cc:1428] Setting key 2fd204c6273da03084af8c9e9cf53db0
I20260927 21:04:25.850674 30587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:25.852854 30587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.190.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.252@KRBTEST.COM" created.
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:25.865207 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.853702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.16.190.254:45121)
0927 21:04:25.853762 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:25.853766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:25.854021 (+ 255us) server_negotiation.cc:407] Connection header received
0927 21:04:25.854576 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:25.854577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:25.854594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:25.854641 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:25.855301 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.855794 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.856857 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.857070 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.857164 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.861892 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:25.861894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:25.861895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:25.861904 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:25.863637 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.864103 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.864105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.864105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.864169 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.864445 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.864446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.864446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.864576 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:25.864647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:25.865016 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 21:04:25.865110 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:04:25.865432 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.853686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45121 (local address 127.0.0.1:53194)
0927 21:04:25.853879 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.853931 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:25.854467 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:25.854719 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:25.854723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:25.855003 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:25.855203 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.855207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.855906 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.855911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.856711 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.856713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.857173 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.857174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.857318 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.857869 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:25.857878 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:25.861772 (+ 3894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:25.863731 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.863741 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.863744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.864012 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.864239 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.864240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.864241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.864359 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.864653 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:25.864660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:25.864855 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 21:04:25.864997 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 21:04:25.865176 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260927 21:04:25.866020 30587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:04:25.875324 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.866385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.16.190.253:39969)
0927 21:04:25.866604 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:25.866609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:25.866628 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:04:25.866656 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:25.866658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:25.866669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:25.866734 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:25.867080 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.867747 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.868892 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.869281 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.869357 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.872399 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:25.872408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:25.872408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:25.872420 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:25.874142 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.874552 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.874555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.874555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.874609 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.874782 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.874782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.874782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.874929 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:25.875027 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:25.875135 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 21:04:25.875240 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":189,"thread_start_us":159,"threads_started":1}
I20260927 21:04:25.875468 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.866326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54994)
0927 21:04:25.866395 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.866405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:25.866446 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:25.866767 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:25.866768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:25.866924 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:25.867000 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.867002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.867837 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.867838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.868703 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.868706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.869363 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.869364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.869421 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.869876 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:25.869881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:25.872266 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:25.874227 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.874228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.874229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.874441 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.874671 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.874672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.874672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.874726 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.875008 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:25.875009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:25.875047 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 21:04:25.875100 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:04:25.875324 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":90}
I20260927 21:04:25.876178 30587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:04:25.876567 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.876445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43482)
0927 21:04:25.876502 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.876546 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 21:04:25.876744 30538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
I20260927 21:04:25.877280 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.877147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43486)
0927 21:04:25.877198 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.877258 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:04:25.877424 30587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
I20260927 21:04:25.880514 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:32791
--webserver_interface=127.16.190.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:25.917434 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } attempt: 3
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:04:25.925110 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.917841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:54996)
0927 21:04:25.917908 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.917922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:25.917978 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:25.918149 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:25.918150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:25.918320 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:25.918379 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.918381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.919087 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.919089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.919615 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:25.919617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.920022 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.920024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:25.920107 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.920615 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:25.920619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:25.921901 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:25.924022 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.924023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.924024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.924209 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.924451 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:25.924451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:25.924452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:25.924507 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:25.924781 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:25.924781 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:25.924882 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 21:04:25.924944 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 21:04:25.925034 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 21:04:25.925192 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.917915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.16.190.253:39969)
0927 21:04:25.918010 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:25.918013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:25.918024 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:04:25.918055 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:25.918056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:25.918075 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:25.918142 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:25.918470 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.918995 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.919709 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:25.919926 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:25.920023 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:25.922003 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:25.922004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:25.922005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:25.922013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:25.923801 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.924296 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.924297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.924297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.924374 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:25.924571 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:25.924572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:25.924572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:25.924718 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:25.924788 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:25.925000 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 21:04:25.925095 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:04:25.925561 30512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:04:25.925946 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.925836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43500)
0927 21:04:25.925888 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.925925 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:04:25.926327 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.926248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43512)
0927 21:04:25.926288 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.926312 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:04:25.926431 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
I20260927 21:04:25.926674 30587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 1
I20260927 21:04:25.927003 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.926898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43526)
0927 21:04:25.926950 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.926977 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:04:25.927304 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.927211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43528)
0927 21:04:25.927261 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.927283 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 21:04:25.927395 30587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
I20260927 21:04:25.957628 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 1
I20260927 21:04:25.958102 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.957951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43536)
0927 21:04:25.958041 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.958073 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 21:04:25.958459 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.958365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43552)
0927 21:04:25.958409 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.958436 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:04:25.958570 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
W20260927 21:04:25.965720 30594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:25.965961 30594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:25.965998 30594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:25.967147 30594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:25.967211 30594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:25.967237 30594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:25.967270 30594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:25.967293 30594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:25.968775 30594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45121,127.16.190.253:39969,127.16.190.252:32791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.190.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:25.969035 30594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:25.969218 30594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:25.974144 30602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:25.974269 30599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:25.974305 30594 server_base.cc:1034] running on GCE node
W20260927 21:04:25.974370 30600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:25.974653 30594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:25.974951 30594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:25.976135 30594 hybrid_clock.cc:648] HybridClock initialized: now 1790543065976122 us; error 45 us; skew 500 ppm
Sep 27 21:04:25 dist-test-slave-6c7s krb5kdc[30431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:25.978638 30594 init.cc:377] Logged in from keytab as kudu/127.16.190.252@KRBTEST.COM (short username kudu)
I20260927 21:04:25.979120 30594 webserver.cc:460] Webserver started at http://127.16.190.252:35161/ using document root <none> and password file <none>
I20260927 21:04:25.979346 30594 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:25.979390 30594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:25.979630 30594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:25.980789 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data/instance:
uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "cb2dab0a195820de7b4a9f6e71b14890"
server_key_iv: "fc3681d7c1835e51a294524a6a54ff57"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.981243 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal/instance:
uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "cb2dab0a195820de7b4a9f6e71b14890"
server_key_iv: "fc3681d7c1835e51a294524a6a54ff57"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.982652 30594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:25.983484 30609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:25.983668 30594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:25.983774 30594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d"
format_stamp: "Formatted at 2026-09-27 21:04:25 on dist-test-slave-6c7s"
server_key: "cb2dab0a195820de7b4a9f6e71b14890"
server_key_iv: "fc3681d7c1835e51a294524a6a54ff57"
server_key_version: "encryptionkey@0"
I20260927 21:04:25.983851 30594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:25.995524 30587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 2
I20260927 21:04:25.996016 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.995882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43562)
0927 21:04:25.995943 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.995986 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:04:25.996510 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:25.996404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43566)
0927 21:04:25.996453 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:25.996483 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 21:04:25.996631 30587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
I20260927 21:04:26.033730 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 2
I20260927 21:04:26.034258 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.034110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43568)
0927 21:04:26.034173 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.034219 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:04:26.034660 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.034564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43576)
0927 21:04:26.034611 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.034638 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:04:26.034770 30512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:32791: Network error: Client connection negotiation failed: client connection to 127.16.190.252:32791: connect: Connection refused (error 111)
I20260927 21:04:26.038761 30594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:26.040747 30594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:26.040970 30594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:26.045174 30594 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.252:32791
I20260927 21:04:26.045241 30661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.252:32791 every 8 connection(s)
I20260927 21:04:26.045627 30594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
I20260927 21:04:26.046064 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30594
I20260927 21:04:26.046175 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790542906842326-17147-0/minicluster-data/master-2/wal/instance
I20260927 21:04:26.046373 17147 external_mini_cluster.cc:1428] Setting key e107812033720af45160b5445b9b62ba
I20260927 21:04:26.046705 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:26.049400 30662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:26.057762 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.047051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.16.190.252:32791)
0927 21:04:26.047227 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.047235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.047261 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:04:26.047889 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.047897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.047949 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.048071 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.049079 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.050184 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.051115 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.051548 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.051676 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.054085 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.054092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.054093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.054099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.055742 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.056153 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.056156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.056157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.056266 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.056565 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.056566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.056566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.056894 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.057003 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.057193 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.057472 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260927 21:04:26.058563 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.050142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.16.190.254:45121)
0927 21:04:26.050186 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.050188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.050400 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:04:26.050429 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.050430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.050441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.050505 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.050926 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.051458 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.052434 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.052623 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.052694 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.054878 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.054884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.054884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.054891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.057044 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.057580 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.057581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.057582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.057635 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.057971 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.057972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.057972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.058103 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.058167 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.058367 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.058452 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:04:26.058943 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.050071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45121 (local address 127.0.0.1:53210)
0927 21:04:26.050313 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.050341 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:26.050388 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:26.050542 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:26.050544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:26.050793 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:26.050851 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.050852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.051550 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.051551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.052335 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.052338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.052716 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.052717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.052796 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.053337 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:26.053353 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:26.054776 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:26.057131 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.057135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.057137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.057507 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.057701 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.057701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.057702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.057776 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.058193 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:26.058198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:26.058248 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:04:26.058658 (+ 410us) client_negotiation.cc:240] Negotiation successful
0927 21:04:26.058810 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260927 21:04:26.059444 30662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:26.066785 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.059803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.16.190.253:39969)
0927 21:04:26.059848 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.059850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.059895 (+ 45us) server_negotiation.cc:407] Connection header received
0927 21:04:26.059976 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.059977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.059985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.060024 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.060267 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.060800 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.061512 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.061708 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.061795 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.063533 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.063535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.063535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.063542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.065660 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.066007 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.066008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.066008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.066067 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.066269 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.066270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.066270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.066384 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.066469 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.066619 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.066691 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:04:26.066785 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.059725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:39969 (local address 127.0.0.1:55008)
0927 21:04:26.059851 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.059861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:26.059909 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:26.060036 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:26.060036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:26.060178 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:26.060212 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.060213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.060940 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.060941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.061425 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.061427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.061794 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.061795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.061864 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.062261 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:26.062264 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:26.063438 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:26.065749 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.065750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.065750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.065931 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.066138 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.066139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.066139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.066196 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.066460 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:26.066461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:26.066505 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:04:26.066593 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:04:26.066690 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 21:04:26.066788 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.058833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.16.190.254:45121)
0927 21:04:26.058871 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.058872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.058880 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:04:26.058909 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.058910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.058917 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.058968 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.059606 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.060285 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.061024 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.061226 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.061297 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.064036 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.064038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.064038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.064044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.065611 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.065980 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.065981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.065982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.066037 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.066240 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.066241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.066241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.066358 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.066559 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.066595 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.066672 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:04:26.067147 30662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:04:26.074106 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.067382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43606)
0927 21:04:26.067493 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.067501 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:26.067548 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:26.067673 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:26.067674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:26.067804 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:26.067830 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.067831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.068478 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.068479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.069028 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.069030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.069430 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.069431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.069480 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.069891 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:26.069896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:26.071148 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:26.072950 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.072951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.072951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.073114 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.073321 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.073322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.073322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.073374 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.073657 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:26.073658 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:26.073708 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 21:04:26.073791 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 21:04:26.073916 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 21:04:26.074112 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.067444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.16.190.252:32791)
0927 21:04:26.067475 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.067477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.067556 (+ 79us) server_negotiation.cc:407] Connection header received
0927 21:04:26.067606 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.067607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.067615 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.067669 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.067904 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.068376 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.069109 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.069324 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.069418 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.071243 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.071244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.071245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.071251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.072846 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.073189 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.073190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.073191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.073250 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.073433 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.073434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.073434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.073583 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.073642 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.073859 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.073970 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":15}
I20260927 21:04:26.074860 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.067397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.16.190.253:39969)
0927 21:04:26.067436 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.067438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.067493 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:04:26.067562 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.067562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.067572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.067626 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.068090 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.068529 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.069250 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.069482 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.069540 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.071868 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.071869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.071869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.071877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.073460 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.073843 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.073844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.073844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.073906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.074128 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.074128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.074129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.074275 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.074469 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.074683 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.074760 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:04:26.076027 30662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: Bootstrap starting.
I20260927 21:04:26.076494 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:04:26.077083 30662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:26.077407 30662 log.cc:824] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:26.078421 30662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: No bootstrap required, opened a new log
I20260927 21:04:26.080790 30662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.081015 30662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:26.081055 30662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1dc2b32e134b6f82b2e8eb73c5ac1d, State: Initialized, Role: FOLLOWER
I20260927 21:04:26.081213 30662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.081560 30662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 21:04:26.083711 30686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:04:26.083772 30686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:04:26.084008 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.084095 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:04:26.086908 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.078174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.16.190.254:45121)
0927 21:04:26.078217 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.078219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.078543 (+ 324us) server_negotiation.cc:407] Connection header received
0927 21:04:26.078625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.078627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.078641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.078695 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.079487 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.080232 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081519 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.081797 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081834 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.083491 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.083492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.083493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.083500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.085367 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.085868 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.085869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.085870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.085927 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.086162 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.086163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.086164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.086320 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.086436 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.086731 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.086811 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:04:26.087041 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.078326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.16.190.252:32791)
0927 21:04:26.078369 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.078371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.078708 (+ 337us) server_negotiation.cc:407] Connection header received
0927 21:04:26.078800 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.078801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.078811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.078839 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.079529 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.080220 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081282 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.081587 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081626 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.083329 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.083330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.083331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.083338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.085270 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.085816 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.085818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.085818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.085879 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.086098 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.086098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.086099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.086299 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.086407 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.086873 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.086954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:04:26.084410 30594 master_runner.cc:186] Error getting master registration for 127.16.190.254:45121: OK, instance_id { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" instance_seqno: 1790543065689730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:04:26.087807 30594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:04:26.087788 30587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 3
I20260927 21:04:26.089932 30592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.078147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.16.190.253:39969)
0927 21:04:26.078188 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.078190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.078593 (+ 403us) server_negotiation.cc:407] Connection header received
0927 21:04:26.078617 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.078618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.078629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.078673 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.079401 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.080131 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081159 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.081437 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.081503 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.084725 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.084727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.084727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.084737 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.086458 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.088731 (+ 2273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.088732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.088733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.088805 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.089087 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.089088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.089089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.089293 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.089447 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.089682 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.089814 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:04:26.095768 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.088045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43636)
0927 21:04:26.088096 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.088107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:26.088146 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:26.088330 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:26.088331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:26.088506 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:26.088562 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.088565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.089399 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.089400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.090200 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.090203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.090685 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.090686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.090742 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.091226 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:26.091231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:26.092639 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:26.094666 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.094668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.094669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.094851 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.095070 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.095071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.095072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.095131 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.095413 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:26.095414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:26.095462 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:04:26.095558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 21:04:26.095667 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:04:26.095768 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.088128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.16.190.252:32791)
0927 21:04:26.088164 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.088166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.088175 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:26.088196 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.088197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.088207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.088265 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.088637 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.089304 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.090279 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.090569 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.090632 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.092735 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.092736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.092737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.092744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.094561 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.094927 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.094928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.094929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.094995 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.095216 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.095217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.095218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.095343 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.095421 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.095584 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.095667 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:04:26.098067 30587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: Bootstrap starting.
I20260927 21:04:26.098891 30587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:26.099236 30587 log.cc:824] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:26.100173 30587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: No bootstrap required, opened a new log
I20260927 21:04:26.102324 30587 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.102481 30587 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:26.102517 30587 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b0e7223bf44c6e935e6860163fa2a3, State: Initialized, Role: FOLLOWER
I20260927 21:04:26.102665 30587 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.102967 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.103014 30587 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:26.103089 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:04:26.105659 30700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:04:26.105712 30700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:04:26.105518 30519 master_runner.cc:186] Error getting master registration for 127.16.190.254:45121: OK, instance_id { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" instance_seqno: 1790543065689730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:04:26.107738 30519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:04:26.133914 30512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } attempt: 3
Sep 27 21:04:26 dist-test-slave-6c7s krb5kdc[30431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:04:26.142472 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.134274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:32791 (local address 127.0.0.1:43648)
0927 21:04:26.134337 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:26.134352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:26.134409 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:26.134529 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:26.134529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:26.134731 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:26.134790 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.134792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.135729 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.135730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.136555 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:26.136557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.137090 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.137091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:26.137154 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.137704 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:26.137710 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:26.139311 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:26.141291 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.141292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.141293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.141528 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.141760 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:26.141761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:26.141761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:26.141835 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:26.142122 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:26.142123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:26.142175 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 21:04:26.142250 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:04:26.142363 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 21:04:26.142478 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:26.134371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.16.190.252:32791)
0927 21:04:26.134415 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:26.134418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:26.134430 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:04:26.134457 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:26.134457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:26.134468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:26.134525 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:26.134878 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.135636 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.136653 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:26.137003 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:26.137081 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:26.139409 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:26.139411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:26.139411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:26.139419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:26.141201 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.141611 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.141612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.141613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.141684 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:26.141903 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:26.141904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:26.141905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:26.142052 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:26.142121 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:26.142291 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 21:04:26.142380 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:04:26.144898 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: Bootstrap starting.
I20260927 21:04:26.145731 30512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:26.146055 30512 log.cc:824] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:26.146994 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: No bootstrap required, opened a new log
I20260927 21:04:26.149147 30512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.149304 30512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:26.149338 30512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d6b8e86d247470688b86b24997e84ec, State: Initialized, Role: FOLLOWER
I20260927 21:04:26.149490 30512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.149843 30512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:26.149811 30703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.150015 30703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:04:26.151713 30714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:04:26.151820 30714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:04:26.154397 30444 master_runner.cc:263] No leader master found from master 1d6b8e86d247470688b86b24997e84ec
I20260927 21:04:26.154461 30444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 21:04:26.187565 30465 tablet.cc:2369] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:26.250121 30703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:26.250274 30703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.250617 30703 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0b0e7223bf44c6e935e6860163fa2a3 (127.16.190.253:39969), 3b1dc2b32e134b6f82b2e8eb73c5ac1d (127.16.190.252:32791)
I20260927 21:04:26.250867 30561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6b8e86d247470688b86b24997e84ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" is_pre_election: true
I20260927 21:04:26.251029 30561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6b8e86d247470688b86b24997e84ec in term 0.
I20260927 21:04:26.251245 30462 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6b8e86d247470688b86b24997e84ec, d0b0e7223bf44c6e935e6860163fa2a3; no voters:
I20260927 21:04:26.251384 30703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:04:26.251448 30703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:04:26.251480 30703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:26.252442 30703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.252424 30636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6b8e86d247470688b86b24997e84ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" is_pre_election: true
I20260927 21:04:26.252563 30703 leader_election.cc:288] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [CANDIDATE]: Term 1 election: Requested vote from peers d0b0e7223bf44c6e935e6860163fa2a3 (127.16.190.253:39969), 3b1dc2b32e134b6f82b2e8eb73c5ac1d (127.16.190.252:32791)
I20260927 21:04:26.252578 30636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d6b8e86d247470688b86b24997e84ec in term 0.
I20260927 21:04:26.252735 30561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6b8e86d247470688b86b24997e84ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b0e7223bf44c6e935e6860163fa2a3"
I20260927 21:04:26.252846 30561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:26.253110 30636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d6b8e86d247470688b86b24997e84ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d"
I20260927 21:04:26.253182 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:26.253754 30561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6b8e86d247470688b86b24997e84ec in term 1.
I20260927 21:04:26.253916 30462 leader_election.cc:302] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d6b8e86d247470688b86b24997e84ec, d0b0e7223bf44c6e935e6860163fa2a3; no voters:
I20260927 21:04:26.253997 30703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:26.254017 30636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d6b8e86d247470688b86b24997e84ec in term 1.
I20260927 21:04:26.254117 30703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [term 1 LEADER]: Becoming Leader. State: Replica: 1d6b8e86d247470688b86b24997e84ec, State: Running, Role: LEADER
I20260927 21:04:26.254205 30703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } }
I20260927 21:04:26.254628 30716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6b8e86d247470688b86b24997e84ec. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.254737 30716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:26.254925 30718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:26.255152 30718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:26.256507 30561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [term 1 FOLLOWER]: Refusing update from remote peer 1d6b8e86d247470688b86b24997e84ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:26.256517 30636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [term 1 FOLLOWER]: Refusing update from remote peer 1d6b8e86d247470688b86b24997e84ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:26.256801 30716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:26.256922 30703 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:26.258140 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6b8e86d247470688b86b24997e84ec. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.258251 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:04:26.258523 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d6b8e86d247470688b86b24997e84ec. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.258616 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:04:26.262277 30716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.262377 30716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:26.262466 30721 mvcc.cc:205] Tried to move back new op lower bound from 7334064399384616960 to 7334064399377965056. Current Snapshot: MvccSnapshot[applied={T|T < 7334064399384616960}]
I20260927 21:04:26.262538 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.262599 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:04:26.263389 30703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.263479 30703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:26.264106 30689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d6b8e86d247470688b86b24997e84ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d6b8e86d247470688b86b24997e84ec" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45121 } } peers { permanent_uuid: "d0b0e7223bf44c6e935e6860163fa2a3" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 39969 } } peers { permanent_uuid: "3b1dc2b32e134b6f82b2e8eb73c5ac1d" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 32791 } } }
I20260927 21:04:26.264201 30689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:04:26.264798 30718 catalog_manager.cc:1347] Generated new cluster ID: 71953a682cf7466da412e566b7995df7
I20260927 21:04:26.264882 30718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:04:26.274647 30718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:04:26.275216 30718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:26.280123 30718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d6b8e86d247470688b86b24997e84ec: Generated new TSK 0
I20260927 21:04:26.280421 30718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:04:27.084263 30686 catalog_manager.cc:1259] Loaded cluster ID: 71953a682cf7466da412e566b7995df7
I20260927 21:04:27.084372 30686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: loading cluster ID for follower catalog manager: success
I20260927 21:04:27.085275 30686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: acquiring CA information for follower catalog manager: success
I20260927 21:04:27.085580 30686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b1dc2b32e134b6f82b2e8eb73c5ac1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:04:27.106174 30700 catalog_manager.cc:1259] Loaded cluster ID: 71953a682cf7466da412e566b7995df7
I20260927 21:04:27.106276 30700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: loading cluster ID for follower catalog manager: success
I20260927 21:04:27.107167 30700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: acquiring CA information for follower catalog manager: success
I20260927 21:04:27.107497 30700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0b0e7223bf44c6e935e6860163fa2a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:04:27.330792 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30444
I20260927 21:04:27.336186 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30519
I20260927 21:04:27.341164 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30594
2026-09-27T21:04:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2946 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:04:27 dist-test-slave-6c7s krb5kdc[30732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:04:27 dist-test-slave-6c7s krb5kdc[30732](info): set up 2 sockets
Sep 27 21:04:27 dist-test-slave-6c7s krb5kdc[30732](info): commencing operation
krb5kdc: starting...
W20260927 21:04:28.445737 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:04:28.577090 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38609
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:38609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:28.661439 30745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:28.661707 30745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:28.661746 30745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:28.663007 30745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:28.663082 30745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:28.663108 30745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:28.663137 30745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:28.663170 30745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:28.664690 30745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:38609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:28.665001 30745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:28.665239 30745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:28.670478 30751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:28.670471 30753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:28.670547 30745 server_base.cc:1034] running on GCE node
W20260927 21:04:28.670471 30750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:28.671072 30745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:28.671358 30745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:28.672533 30745 hybrid_clock.cc:648] HybridClock initialized: now 1790543068672509 us; error 44 us; skew 500 ppm
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543068, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:28.675577 30745 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:28.676131 30745 webserver.cc:460] Webserver started at http://127.16.190.254:40519/ using document root <none> and password file <none>
I20260927 21:04:28.676386 30745 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:28.676473 30745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:28.676635 30745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:28.677886 30745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "a43c416c37814f5cab2c0ec8bc8c46a2"
format_stamp: "Formatted at 2026-09-27 21:04:28 on dist-test-slave-6c7s"
server_key: "6897ae02c961b5111c4cc7894844674d"
server_key_iv: "baf6eea5240f75a7c36ca2c459650ba2"
server_key_version: "encryptionkey@0"
I20260927 21:04:28.678280 30745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "a43c416c37814f5cab2c0ec8bc8c46a2"
format_stamp: "Formatted at 2026-09-27 21:04:28 on dist-test-slave-6c7s"
server_key: "6897ae02c961b5111c4cc7894844674d"
server_key_iv: "baf6eea5240f75a7c36ca2c459650ba2"
server_key_version: "encryptionkey@0"
I20260927 21:04:28.679719 30745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:28.680497 30760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:28.680689 30745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:04:28.680790 30745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a43c416c37814f5cab2c0ec8bc8c46a2"
format_stamp: "Formatted at 2026-09-27 21:04:28 on dist-test-slave-6c7s"
server_key: "6897ae02c961b5111c4cc7894844674d"
server_key_iv: "baf6eea5240f75a7c36ca2c459650ba2"
server_key_version: "encryptionkey@0"
I20260927 21:04:28.680932 30745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:28.701996 30745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:28.704464 30745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:28.704702 30745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:28.708791 30745 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:38609
I20260927 21:04:28.708904 30812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:38609 every 8 connection(s)
I20260927 21:04:28.709275 30745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:28.710194 30813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:28.712652 30813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2: Bootstrap starting.
I20260927 21:04:28.712760 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30745
I20260927 21:04:28.712885 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:04:28.713124 17147 external_mini_cluster.cc:1428] Setting key 42bd8428e34b9f3b3666eda3626e4d67
I20260927 21:04:28.713469 30813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:28.713796 30813 log.cc:824] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:28.714632 30813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2: No bootstrap required, opened a new log
I20260927 21:04:28.716509 30813 raft_consensus.cc:348] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } }
I20260927 21:04:28.716657 30813 raft_consensus.cc:374] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:28.716679 30813 raft_consensus.cc:729] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43c416c37814f5cab2c0ec8bc8c46a2, State: Initialized, Role: FOLLOWER
I20260927 21:04:28.716854 30813 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } }
I20260927 21:04:28.716944 30813 raft_consensus.cc:388] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:28.716977 30813 raft_consensus.cc:482] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:28.717029 30813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:28.717648 30813 raft_consensus.cc:504] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } }
I20260927 21:04:28.717774 30813 leader_election.cc:302] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43c416c37814f5cab2c0ec8bc8c46a2; no voters:
I20260927 21:04:28.717969 30813 leader_election.cc:288] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:04:28.718019 30818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:28.718195 30818 raft_consensus.cc:686] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [term 1 LEADER]: Becoming Leader. State: Replica: a43c416c37814f5cab2c0ec8bc8c46a2, State: Running, Role: LEADER
I20260927 21:04:28.718281 30813 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:28.718297 30818 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } }
I20260927 21:04:28.718680 30818 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43c416c37814f5cab2c0ec8bc8c46a2. Latest consensus state: current_term: 1 leader_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } } }
I20260927 21:04:28.718780 30818 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:28.718689 30820 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43c416c37814f5cab2c0ec8bc8c46a2" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38609 } } }
I20260927 21:04:28.718945 30820 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:28.719027 30825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:28.719564 30825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:28.721323 30825 catalog_manager.cc:1347] Generated new cluster ID: 134f7a24b4b94ebcafa45c4b89636a28
I20260927 21:04:28.721374 30825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:04:28.724925 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:28.713629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.16.190.254:38609)
0927 21:04:28.713858 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:28.713870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:28.713908 (+ 38us) server_negotiation.cc:407] Connection header received
0927 21:04:28.714577 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:28.714586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:28.714642 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:28.714819 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:28.715460 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:28.716349 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:28.717430 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:28.717988 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:28.719950 (+ 1962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:28.721142 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:28.721146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:28.721146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:28.721160 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:28.723178 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:28.723592 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:28.723595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:28.723595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:28.723712 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:28.723932 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:28.723933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:28.723934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:28.724261 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:28.724434 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:28.724514 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 21:04:28.724662 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260927 21:04:28.725875 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:04:28.727564 30825 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:04:28.728071 30825 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:28.734694 30825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a43c416c37814f5cab2c0ec8bc8c46a2: Generated new TSK 0
I20260927 21:04:28.734925 30825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:04:28.780242 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30745
2026-09-27T21:04:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1436 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30840](info): set up 2 sockets
Sep 27 21:04:28 dist-test-slave-6c7s krb5kdc[30840](info): commencing operation
krb5kdc: starting...
W20260927 21:04:29.829957 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:04:29 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:04:29.957460 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:30.042856 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:30.043103 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:30.043143 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:30.044286 30853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:30.044350 30853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:30.044377 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:30.044409 30853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:30.044433 30853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:30.045986 30853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:30.046263 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:30.046490 30853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:30.051378 30861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.051384 30859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.051384 30858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.051721 30853 server_base.cc:1034] running on GCE node
I20260927 21:04:30.051954 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:30.052312 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:30.053491 30853 hybrid_clock.cc:648] HybridClock initialized: now 1790543070053464 us; error 36 us; skew 500 ppm
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:30.063086 30853 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:30.063589 30853 webserver.cc:460] Webserver started at http://127.16.190.254:46149/ using document root <none> and password file <none>
I20260927 21:04:30.063841 30853 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:30.063913 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:30.064077 30853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:30.065321 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "a648c63003a648cf9a5a1c95551b1f98"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "6c846a9ba47b497712c1fdf91f85ee09"
server_key_iv: "bb9391824c285395b5f522e91c2306f6"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.065708 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "a648c63003a648cf9a5a1c95551b1f98"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "6c846a9ba47b497712c1fdf91f85ee09"
server_key_iv: "bb9391824c285395b5f522e91c2306f6"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.067186 30853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:04:30.068089 30868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.068305 30853 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:30.068419 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a648c63003a648cf9a5a1c95551b1f98"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "6c846a9ba47b497712c1fdf91f85ee09"
server_key_iv: "bb9391824c285395b5f522e91c2306f6"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.068506 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:30.087927 30853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:30.090498 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:30.090698 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:30.094926 30853 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36733
I20260927 21:04:30.094969 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36733 every 8 connection(s)
I20260927 21:04:30.095394 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:30.096467 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:30.098942 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap starting.
I20260927 21:04:30.099550 30921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:30.099865 30921 log.cc:824] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:30.100708 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: No bootstrap required, opened a new log
I20260927 21:04:30.102547 30921 raft_consensus.cc:348] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.102710 30921 raft_consensus.cc:374] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:30.102735 30921 raft_consensus.cc:729] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Initialized, Role: FOLLOWER
I20260927 21:04:30.102911 30921 consensus_queue.cc:260] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.103013 30921 raft_consensus.cc:388] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:30.103082 30921 raft_consensus.cc:482] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:30.103169 30921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:30.103205 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30853
I20260927 21:04:30.103336 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:04:30.103551 17147 external_mini_cluster.cc:1428] Setting key 46ae40b18e51635d38ebd7d335afc423
I20260927 21:04:30.104415 30921 raft_consensus.cc:504] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.104552 30921 leader_election.cc:302] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a648c63003a648cf9a5a1c95551b1f98; no voters:
I20260927 21:04:30.104754 30921 leader_election.cc:288] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:04:30.104775 30926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:30.104921 30926 raft_consensus.cc:686] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 LEADER]: Becoming Leader. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Running, Role: LEADER
I20260927 21:04:30.105070 30926 consensus_queue.cc:237] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.105083 30921 sys_catalog.cc:564] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:30.106493 30927 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:30.106578 30927 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:30.106724 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a648c63003a648cf9a5a1c95551b1f98. Latest consensus state: current_term: 1 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:30.106772 30941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:30.106804 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:30.107717 30941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:30.109700 30941 catalog_manager.cc:1347] Generated new cluster ID: 9ba86eae9aff4205a978d543325e9abd
I20260927 21:04:30.109764 30941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:30.115545 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.104096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.16.190.254:36733)
0927 21:04:30.104276 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.104285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.104311 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:04:30.104413 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.104418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.104461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.104620 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:30.105488 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.106989 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.107818 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.108253 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.108391 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.111566 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:30.111572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:30.111572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:30.111585 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:30.113708 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.114185 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.114188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.114188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.114308 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.114567 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.114568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.114568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.114875 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:30.115021 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.115172 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.115315 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260927 21:04:30.117341 30941 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:04:30.118131 30941 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:30.122931 30941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Generated new TSK 0
I20260927 21:04:30.123159 30941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:04:30.156598 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:30.240266 30947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:30.240530 30947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:30.240569 30947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:30.241899 30947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:30.241984 30947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:30.242030 30947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:30.243676 30947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:30.243986 30947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:30.244300 30947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:30.250855 30955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.250871 30953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.250885 30952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.251358 30947 server_base.cc:1034] running on GCE node
I20260927 21:04:30.251600 30947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:30.251929 30947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:30.253108 30947 hybrid_clock.cc:648] HybridClock initialized: now 1790543070253096 us; error 41 us; skew 500 ppm
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:30.256409 30947 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:30.256983 30947 webserver.cc:460] Webserver started at http://127.16.190.193:41325/ using document root <none> and password file <none>
I20260927 21:04:30.257236 30947 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:30.257285 30947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:30.257486 30947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:30.258666 30947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "bbd344e03f4c40f0e9f8f9019d67639c"
server_key_iv: "f66935e35a7b67e47375fc076234193d"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.259065 30947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "bbd344e03f4c40f0e9f8f9019d67639c"
server_key_iv: "f66935e35a7b67e47375fc076234193d"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.260537 30947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:30.261492 30962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.261721 30947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:30.261837 30947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "bbd344e03f4c40f0e9f8f9019d67639c"
server_key_iv: "f66935e35a7b67e47375fc076234193d"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.261924 30947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:30.287818 30947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:30.290112 30947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:30.290295 30947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:30.290580 30947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:30.290916 30947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:30.290982 30947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.291033 30947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:30.291060 30947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.297526 30947 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:45097
I20260927 21:04:30.297575 31075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:45097 every 8 connection(s)
I20260927 21:04:30.297953 30947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:30.302423 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 30947
I20260927 21:04:30.302515 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:04:30.302736 17147 external_mini_cluster.cc:1428] Setting key 91f96eca15666adac3d2d32bb74d49b6
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:30.309240 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.298830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:58737 (local address 127.16.190.254:36733)
0927 21:04:30.298903 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.298905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.299168 (+ 263us) server_negotiation.cc:407] Connection header received
0927 21:04:30.299735 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.299736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.299761 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.299851 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:30.300931 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.301858 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.303108 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.303342 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.303430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.305931 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:30.305933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:30.305933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:30.305943 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:30.307600 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.308154 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.308154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.308155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.308205 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.308505 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.308505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.308506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.308632 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:30.308683 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.309101 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.309173 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:04:30.309513 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.298801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:58737)
0927 21:04:30.298989 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.299033 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.299614 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.299928 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.299932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.300543 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:30.300743 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.300748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.301981 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.301982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.303003 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.303007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.303443 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.303444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.303574 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.304167 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:30.304176 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:30.305842 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:30.307688 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.307696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.307699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.308065 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.308273 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.308273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.308274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.308432 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.308702 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:30.308709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:30.308940 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.309073 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.309259 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260927 21:04:30.310102 31076 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:30.310205 31076 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:30.310417 31076 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:30.311056 30885 ts_manager.cc:194] Registered new tserver with Master: 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097)
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:04:30.311951 30885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:58737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:04:30.333923 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:30.418393 31081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:30.418736 31081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:30.418785 31081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:30.420454 31081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:30.420509 31081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:30.420539 31081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:04:30.422134 31081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:30.422412 31081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:30.422678 31081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:30.428522 31089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.428511 31086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.428512 31087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.428905 31081 server_base.cc:1034] running on GCE node
I20260927 21:04:30.429109 31081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:30.429335 31081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:30.430538 31081 hybrid_clock.cc:648] HybridClock initialized: now 1790543070430491 us; error 38 us; skew 500 ppm
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:30.433362 31081 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:30.433902 31081 webserver.cc:460] Webserver started at http://127.16.190.194:38631/ using document root <none> and password file <none>
I20260927 21:04:30.434149 31081 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:30.434229 31081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:30.434369 31081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:30.435462 31081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "fffc8e1c33624938a54c0e2a7a581047"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "3e2d9c2487f3c2bd42ddea555aec3aaf"
server_key_iv: "4a35fff433a85b67746ebda58e227d77"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.435849 31081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "fffc8e1c33624938a54c0e2a7a581047"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "3e2d9c2487f3c2bd42ddea555aec3aaf"
server_key_iv: "4a35fff433a85b67746ebda58e227d77"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.437526 31081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:30.438498 31096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.438849 31081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:30.438958 31081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "fffc8e1c33624938a54c0e2a7a581047"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "3e2d9c2487f3c2bd42ddea555aec3aaf"
server_key_iv: "4a35fff433a85b67746ebda58e227d77"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.439054 31081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:30.461591 31081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:30.464285 31081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:30.464506 31081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:30.464775 31081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:30.465332 31081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:30.465416 31081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.465464 31081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:30.465494 31081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.471639 31081 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:34109
I20260927 21:04:30.471688 31209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:34109 every 8 connection(s)
I20260927 21:04:30.472043 31081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:30.479615 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31081
I20260927 21:04:30.479733 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:04:30.479965 17147 external_mini_cluster.cc:1428] Setting key 1407b60eadd9e89768f7c07f70c61085
I20260927 21:04:30.482308 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.472755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:51035 (local address 127.16.190.254:36733)
0927 21:04:30.472828 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.472831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.473018 (+ 187us) server_negotiation.cc:407] Connection header received
0927 21:04:30.473589 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.473591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.473604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.473657 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:30.474682 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.475201 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.476237 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.476449 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.476533 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.478851 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:30.478853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:30.478853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:30.478860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:30.480596 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.481128 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.481130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.481130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.481196 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.481475 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.481476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.481476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.481600 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:30.481674 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.482025 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.482132 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:04:30.482568 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.472708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:51035)
0927 21:04:30.472882 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.472931 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.473469 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.473746 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.473756 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.474341 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:30.474572 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.474578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.475312 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.475313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.476138 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.476140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.476549 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.476550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.476682 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.477272 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:30.477281 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:30.478738 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:30.480686 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.480695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.480697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.481030 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.481267 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.481268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.481268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.481388 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.481676 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:30.481684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:30.481836 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.481950 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.482260 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260927 21:04:30.483175 31210 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:30.483296 31210 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:30.483505 31210 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:30.484048 30885 ts_manager.cc:194] Registered new tserver with Master: fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
I20260927 21:04:30.484531 30885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:51035
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:04:30.507848 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:30.590387 31215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:30.590662 31215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:30.590703 31215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:30.591905 31215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:30.591989 31215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:30.592036 31215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:30.593636 31215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:30.593945 31215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:30.594204 31215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:30.599587 31221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.599638 31215 server_base.cc:1034] running on GCE node
W20260927 21:04:30.599602 31220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.599604 31223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.599924 31215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:30.600222 31215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:30.601387 31215 hybrid_clock.cc:648] HybridClock initialized: now 1790543070601363 us; error 36 us; skew 500 ppm
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:30.604074 31215 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:30.604532 31215 webserver.cc:460] Webserver started at http://127.16.190.195:43627/ using document root <none> and password file <none>
I20260927 21:04:30.604787 31215 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:30.604890 31215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:30.605050 31215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:30.606395 31215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "3eb90ff82d924946b51e53f002cae203"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "46376d32a8b87f57e00aad85558eaf6c"
server_key_iv: "357624576c00ec9a8c537c5864452faf"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.606819 31215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "3eb90ff82d924946b51e53f002cae203"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "46376d32a8b87f57e00aad85558eaf6c"
server_key_iv: "357624576c00ec9a8c537c5864452faf"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.608249 31215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:30.609138 31230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.609328 31215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:30.609496 31215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3eb90ff82d924946b51e53f002cae203"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "46376d32a8b87f57e00aad85558eaf6c"
server_key_iv: "357624576c00ec9a8c537c5864452faf"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.609586 31215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:30.628149 31215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:30.630635 31215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:30.630818 31215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:30.631081 31215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:30.631424 31215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:30.631479 31215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.631524 31215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:30.631565 31215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.637929 31215 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:43197
I20260927 21:04:30.637977 31343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:43197 every 8 connection(s)
I20260927 21:04:30.638351 31215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:04:30.643124 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31215
I20260927 21:04:30.643214 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:04:30.643455 17147 external_mini_cluster.cc:1428] Setting key 6c1d47188292557dca2087af7fa48546
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:30.648597 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.639216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:56249 (local address 127.16.190.254:36733)
0927 21:04:30.639296 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.639299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.639544 (+ 245us) server_negotiation.cc:407] Connection header received
0927 21:04:30.640078 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.640079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.640092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.640149 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:30.641277 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.642157 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.643302 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.643609 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.643697 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.646102 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:30.646104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:30.646104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:30.646113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:30.647381 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.647772 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.647773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.647774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.647823 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.648030 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.648031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.648031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.648152 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:30.648201 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.648458 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.648524 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 21:04:30.648756 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.639207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:56249)
0927 21:04:30.639366 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.639452 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.639975 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.640229 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.640233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.640861 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:30.641153 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.641160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.642237 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.642238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.643202 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.643205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.643703 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.643704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.643875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.644526 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:30.644534 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:30.645992 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:30.647471 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.647477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.647479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.647697 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.647879 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:30.647880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:30.647880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:30.647975 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:30.648211 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:30.648216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:30.648337 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.648431 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.648553 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260927 21:04:30.649248 31344 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:30.649363 31344 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:30.649569 31344 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:30.649952 30885 ts_manager.cc:194] Registered new tserver with Master: 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197)
I20260927 21:04:30.650287 30885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:56249
I20260927 21:04:30.655181 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:04:30.662743 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.656488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.16.190.254:36733)
0927 21:04:30.656560 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.656562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.656748 (+ 186us) server_negotiation.cc:407] Connection header received
0927 21:04:30.656774 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.656775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.656785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.656917 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:30.657624 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.658118 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.658855 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.659082 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.659185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.660331 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:30.660332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:30.660333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:30.660339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:30.661785 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.662086 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.662087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.662087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.662137 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:30.662301 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:30.662301 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:30.662301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:30.662439 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:30.662513 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.662597 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.662654 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:04:30.664237 30885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:04:30.666122 30885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:04:30.672797 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.16.190.195:43197)
0927 21:04:30.669018 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.669023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.669034 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:04:30.669076 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.669078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.669094 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.669392 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:30.669471 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670143 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.671561 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.672302 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.672431 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.672535 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.672635 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.672682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":85,"threads_started":1}
I20260927 21:04:30.672782 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43197 (local address 127.0.0.1:58002)
0927 21:04:30.668659 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.668701 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.668775 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.669215 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.669219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.669222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:30.669307 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.669309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670249 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.670251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.671470 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.671472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.672446 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.672447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.672533 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.672551 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.672623 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.672695 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":56,"threads_started":1}
I20260927 21:04:30.672801 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34109 (local address 127.0.0.1:45090)
0927 21:04:30.668724 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.668733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.668776 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.669292 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.669293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.669296 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:30.669360 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.669361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670778 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.670779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.671849 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.671851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.672594 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.672595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.672668 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.672671 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.672701 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.672743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":26,"thread_start_us":96,"threads_started":1}
I20260927 21:04:30.673223 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.16.190.194:34109)
0927 21:04:30.669106 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.669112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.669122 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:04:30.669166 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.669168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.669184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.669524 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:30.669605 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670637 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.671937 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.672496 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.673061 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.673105 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.673127 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.673163 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":119,"threads_started":1}
I20260927 21:04:30.673774 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:45097 (local address 127.0.0.1:55990)
0927 21:04:30.668914 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.668921 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.668957 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.669292 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.669293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.669298 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:30.669433 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.669434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670751 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.670752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.671868 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.671871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.673510 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.673511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.673584 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.673589 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.673649 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.673706 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":129,"spinlock_wait_cycles":1280,"thread_start_us":106,"threads_started":1}
I20260927 21:04:30.673686 31278 tablet_service.cc:1467] Processing CreateTablet for tablet 8b093ebf176340d1a59251cddcffe2ec (DEFAULT_TABLE table=test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:30.674023 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.668924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.16.190.193:45097)
0927 21:04:30.669133 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.669137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.669146 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:30.669182 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.669184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.669198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.669330 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:30.669487 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.670617 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.671963 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.672707 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.672775 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.672853 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.673885 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.673934 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260927 21:04:30.673951 31144 tablet_service.cc:1467] Processing CreateTablet for tablet 8b093ebf176340d1a59251cddcffe2ec (DEFAULT_TABLE table=test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:30.674077 31278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b093ebf176340d1a59251cddcffe2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:30.674283 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b093ebf176340d1a59251cddcffe2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:30.676877 31361 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Bootstrap starting.
I20260927 21:04:30.677211 31362 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Bootstrap starting.
I20260927 21:04:30.677558 31361 tablet_bootstrap.cc:654] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:30.677841 31362 tablet_bootstrap.cc:654] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:30.677742 31010 tablet_service.cc:1467] Processing CreateTablet for tablet 8b093ebf176340d1a59251cddcffe2ec (DEFAULT_TABLE table=test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:30.678118 31010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b093ebf176340d1a59251cddcffe2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:30.678124 31361 log.cc:824] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:30.678421 31362 log.cc:824] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:30.679013 31361 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: No bootstrap required, opened a new log
I20260927 21:04:30.679085 31361 ts_tablet_manager.cc:1397] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:04:30.681143 31361 raft_consensus.cc:348] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.681330 31361 raft_consensus.cc:374] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:30.681375 31362 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: No bootstrap required, opened a new log
I20260927 21:04:30.681391 31361 raft_consensus.cc:729] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb90ff82d924946b51e53f002cae203, State: Initialized, Role: FOLLOWER
I20260927 21:04:30.681434 31362 ts_tablet_manager.cc:1397] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 21:04:30.681557 31361 consensus_queue.cc:260] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.681615 31365 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap starting.
I20260927 21:04:30.681855 31361 ts_tablet_manager.cc:1428] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 21:04:30.682004 31344 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
I20260927 21:04:30.682260 31365 tablet_bootstrap.cc:654] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:30.682808 31365 log.cc:824] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:30.683684 31365 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: No bootstrap required, opened a new log
I20260927 21:04:30.683761 31365 ts_tablet_manager.cc:1397] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:04:30.683687 31362 raft_consensus.cc:348] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.683867 31362 raft_consensus.cc:374] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:30.683910 31362 raft_consensus.cc:729] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc8e1c33624938a54c0e2a7a581047, State: Initialized, Role: FOLLOWER
I20260927 21:04:30.684022 31362 consensus_queue.cc:260] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.684327 31210 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
I20260927 21:04:30.684744 31362 ts_tablet_manager.cc:1428] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:04:30.685662 31365 raft_consensus.cc:348] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.685782 31365 raft_consensus.cc:374] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:30.685803 31365 raft_consensus.cc:729] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a97e87d9aae4f30b09b2eab60fdbd62, State: Initialized, Role: FOLLOWER
I20260927 21:04:30.685905 31365 consensus_queue.cc:260] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.686131 31365 ts_tablet_manager.cc:1428] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:04:30.686203 31076 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
W20260927 21:04:30.722651 31211 tablet.cc:2369] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:30.732308 31369 raft_consensus.cc:482] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:30.732548 31369 raft_consensus.cc:504] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.733026 31369 leader_election.cc:288] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
I20260927 21:04:30.736594 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.733321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:38705 (local address 127.16.190.194:34109)
0927 21:04:30.733374 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.733377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.733400 (+ 23us) server_negotiation.cc:407] Connection header received
0927 21:04:30.733428 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.733429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.733438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.733481 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:30.733735 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.734400 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.735619 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.736235 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.736293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.736300 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.736507 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.736538 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:04:30.736631 31078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.733233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34109 (local address 127.16.190.193:38705)
0927 21:04:30.733291 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.733304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.733352 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.733518 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.733520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.733522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:30.733623 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.733625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.734505 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.734507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.735505 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.735508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.736332 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.736333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.736416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.736421 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.736485 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.736526 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:04:30.737025 31164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b093ebf176340d1a59251cddcffe2ec" candidate_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffc8e1c33624938a54c0e2a7a581047" is_pre_election: true
I20260927 21:04:30.737105 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.733453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43197 (local address 127.16.190.193:45523)
0927 21:04:30.733669 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.733699 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:30.733760 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:30.733911 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:30.733911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:30.733913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:30.733981 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.733982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.734888 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.734889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.736056 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:30.736060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.736872 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.736873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:30.736947 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.736952 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:30.737013 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:04:30.737052 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":144,"threads_started":1}
I20260927 21:04:30.737159 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.733498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:45523 (local address 127.16.190.195:43197)
0927 21:04:30.733550 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.733552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.733754 (+ 202us) server_negotiation.cc:407] Connection header received
0927 21:04:30.733831 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.733832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.733841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.733886 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:30.734089 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.734759 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.736169 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.736770 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.736875 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.736885 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.737045 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.737090 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:04:30.737344 31164 raft_consensus.cc:2454] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a97e87d9aae4f30b09b2eab60fdbd62 in term 0.
I20260927 21:04:30.737673 30964 leader_election.cc:302] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62, fffc8e1c33624938a54c0e2a7a581047; no voters:
I20260927 21:04:30.737669 31298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b093ebf176340d1a59251cddcffe2ec" candidate_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb90ff82d924946b51e53f002cae203" is_pre_election: true
I20260927 21:04:30.737818 31298 raft_consensus.cc:2454] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a97e87d9aae4f30b09b2eab60fdbd62 in term 0.
I20260927 21:04:30.737841 31369 raft_consensus.cc:2781] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:04:30.737913 31369 raft_consensus.cc:482] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:04:30.737989 31369 raft_consensus.cc:3037] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:30.738739 31369 raft_consensus.cc:504] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.738888 31369 leader_election.cc:288] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
I20260927 21:04:30.739009 31298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b093ebf176340d1a59251cddcffe2ec" candidate_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb90ff82d924946b51e53f002cae203"
I20260927 21:04:30.739038 31164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b093ebf176340d1a59251cddcffe2ec" candidate_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffc8e1c33624938a54c0e2a7a581047"
I20260927 21:04:30.739090 31298 raft_consensus.cc:3037] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:30.739104 31164 raft_consensus.cc:3037] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:30.739851 31298 raft_consensus.cc:2454] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a97e87d9aae4f30b09b2eab60fdbd62 in term 1.
I20260927 21:04:30.739854 31164 raft_consensus.cc:2454] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a97e87d9aae4f30b09b2eab60fdbd62 in term 1.
I20260927 21:04:30.740049 30965 leader_election.cc:302] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb90ff82d924946b51e53f002cae203, 7a97e87d9aae4f30b09b2eab60fdbd62; no voters:
I20260927 21:04:30.740160 31369 raft_consensus.cc:2781] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:30.740334 31369 raft_consensus.cc:686] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 LEADER]: Becoming Leader. State: Replica: 7a97e87d9aae4f30b09b2eab60fdbd62, State: Running, Role: LEADER
I20260927 21:04:30.740434 31369 consensus_queue.cc:237] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:30.741346 30885 catalog_manager.cc:5495] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193). New cstate: current_term: 1 leader_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } health_report { overall_health: UNKNOWN } } }
I20260927 21:04:30.765098 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.762592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.16.190.193:45097)
0927 21:04:30.762677 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:30.762678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:30.762687 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:30.762724 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:30.762725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:30.762743 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:30.762827 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:04:30.762946 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.763441 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.764190 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:30.764401 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:30.764461 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:30.764635 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:04:30.764760 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:04:30.764982 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:30.764998 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 21:04:30.765018 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 21:04:30.767130 31164 raft_consensus.cc:1261] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Refusing update from remote peer 7a97e87d9aae4f30b09b2eab60fdbd62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:30.767189 31298 raft_consensus.cc:1261] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Refusing update from remote peer 7a97e87d9aae4f30b09b2eab60fdbd62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:30.767587 31371 consensus_queue.cc:1035] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:30.767719 31369 consensus_queue.cc:1035] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:30.770255 31374 mvcc.cc:205] Tried to move back new op lower bound from 7334064417859133440 to 7334064417753419776. Current Snapshot: MvccSnapshot[applied={T|T < 7334064417859133440}]
I20260927 21:04:30.770422 31375 mvcc.cc:205] Tried to move back new op lower bound from 7334064417859133440 to 7334064417753419776. Current Snapshot: MvccSnapshot[applied={T|T < 7334064417859133440}]
I20260927 21:04:30.772497 30885 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:30.772596 30885 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:30.773383 30885 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Sending DeleteTablet for 3 replicas of tablet 8b093ebf176340d1a59251cddcffe2ec
I20260927 21:04:30.773799 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30947
I20260927 21:04:30.773806 31144 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b093ebf176340d1a59251cddcffe2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:30 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:45090
I20260927 21:04:30.774338 31278 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b093ebf176340d1a59251cddcffe2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:30 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58002
I20260927 21:04:30.774583 31384 tablet_replica.cc:330] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: stopping tablet replica
I20260927 21:04:30.774686 31384 raft_consensus.cc:2229] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:30.774791 31384 raft_consensus.cc:2258] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:30.774838 31383 tablet_replica.cc:330] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: stopping tablet replica
I20260927 21:04:30.774933 31383 raft_consensus.cc:2229] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:30.775185 31383 raft_consensus.cc:2258] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:30.775281 31384 ts_tablet_manager.cc:1905] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:30.777019 31384 ts_tablet_manager.cc:1918] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:30.777097 31384 log.cc:1196] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/8b093ebf176340d1a59251cddcffe2ec
I20260927 21:04:30.777410 31384 ts_tablet_manager.cc:1939] T 8b093ebf176340d1a59251cddcffe2ec P 3eb90ff82d924946b51e53f002cae203: Deleting consensus metadata
I20260927 21:04:30.777396 31383 ts_tablet_manager.cc:1905] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:30.777835 30871 catalog_manager.cc:4841] TS 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): tablet 8b093ebf176340d1a59251cddcffe2ec (table test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]) successfully deleted
I20260927 21:04:30.779139 31383 ts_tablet_manager.cc:1918] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:30.779232 31383 log.cc:1196] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/8b093ebf176340d1a59251cddcffe2ec
I20260927 21:04:30.779527 31383 ts_tablet_manager.cc:1939] T 8b093ebf176340d1a59251cddcffe2ec P fffc8e1c33624938a54c0e2a7a581047: Deleting consensus metadata
I20260927 21:04:30.780014 30870 catalog_manager.cc:4841] TS fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): tablet 8b093ebf176340d1a59251cddcffe2ec (table test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]) successfully deleted
W20260927 21:04:30.781308 30869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:45097 (error 108)
I20260927 21:04:30.781858 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:30.781731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:45097 (local address 127.0.0.1:56004)
0927 21:04:30.781791 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:30.781828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:04:30.781893 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31081
W20260927 21:04:30.782024 30869 catalog_manager.cc:4584] TS 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b093ebf176340d1a59251cddcffe2ec: Network error: Client connection negotiation failed: client connection to 127.16.190.193:45097: connect: Connection refused (error 111)
I20260927 21:04:30.787258 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31215
I20260927 21:04:30.791827 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 30853
I20260927 21:04:30.797598 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--webserver_interface=127.16.190.254
--webserver_port=46149
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:30.888581 31386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:30.888845 31386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:30.888872 31386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:30.890117 31386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:30.890168 31386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:30.890182 31386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:30.890195 31386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:30.890208 31386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:30.891754 31386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=46149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:30.892030 31386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:30.892273 31386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:30.906855 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.906849 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:30.906852 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:30.907290 31386 server_base.cc:1034] running on GCE node
I20260927 21:04:30.907567 31386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:30.907856 31386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:30.908998 31386 hybrid_clock.cc:648] HybridClock initialized: now 1790543070908976 us; error 37 us; skew 500 ppm
Sep 27 21:04:30 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:30.911688 31386 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:30.912176 31386 webserver.cc:460] Webserver started at http://127.16.190.254:46149/ using document root <none> and password file <none>
I20260927 21:04:30.912389 31386 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:30.912449 31386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:30.913899 31386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:30.914484 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:30.914660 31386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:04:30.914767 31386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a648c63003a648cf9a5a1c95551b1f98"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "6c846a9ba47b497712c1fdf91f85ee09"
server_key_iv: "bb9391824c285395b5f522e91c2306f6"
server_key_version: "encryptionkey@0"
I20260927 21:04:30.915184 31386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:30.928869 31386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:30.931348 31386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:30.931566 31386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:30.935974 31386 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36733
I20260927 21:04:30.936005 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36733 every 8 connection(s)
I20260927 21:04:30.936409 31386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:30.938210 31455 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:04:30.938865 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap starting.
I20260927 21:04:30.943190 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31386
I20260927 21:04:30.943477 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:45097
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=41325
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:30.948170 31455 log.cc:824] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:30.951102 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:04:30.951495 31455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap complete.
I20260927 21:04:30.955569 31455 raft_consensus.cc:348] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.955945 31455 raft_consensus.cc:729] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Initialized, Role: FOLLOWER
I20260927 21:04:30.956112 31455 consensus_queue.cc:260] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.956176 31455 raft_consensus.cc:388] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:30.956218 31455 raft_consensus.cc:482] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:30.956264 31455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:04:30.957731 31455 raft_consensus.cc:504] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.957846 31455 leader_election.cc:302] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a648c63003a648cf9a5a1c95551b1f98; no voters:
I20260927 21:04:30.958007 31455 leader_election.cc:288] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:04:30.958122 31459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:04:30.958253 31455 sys_catalog.cc:564] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:30.958294 31459 raft_consensus.cc:686] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 LEADER]: Becoming Leader. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Running, Role: LEADER
I20260927 21:04:30.958462 31459 consensus_queue.cc:237] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:30.958758 31459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a648c63003a648cf9a5a1c95551b1f98. Latest consensus state: current_term: 2 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:30.958856 31459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:30.958758 31461 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:30.958995 31461 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:30.959249 31468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:30.960109 31468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:30.960223 31468 catalog_manager.cc:1259] Loaded cluster ID: 9ba86eae9aff4205a978d543325e9abd
I20260927 21:04:30.960289 31468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:04:30.960731 31468 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 21:04:30.963478 31468 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 21:04:30.965497 31468 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:30.965716 31468 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Loaded TSK: 0
I20260927 21:04:30.965950 31468 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:04:31.048113 31457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:31.048339 31457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:31.048364 31457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:31.049700 31457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:31.049744 31457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:31.049772 31457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:31.051244 31457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:45097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=41325
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:31.051499 31457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:31.051759 31457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:31.058008 31482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.058001 31483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.058115 31485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.058310 31457 server_base.cc:1034] running on GCE node
I20260927 21:04:31.058531 31457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:31.058840 31457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:31.060026 31457 hybrid_clock.cc:648] HybridClock initialized: now 1790543071060004 us; error 55 us; skew 500 ppm
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:31.063126 31457 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:31.063573 31457 webserver.cc:460] Webserver started at http://127.16.190.193:41325/ using document root <none> and password file <none>
I20260927 21:04:31.063805 31457 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:31.063887 31457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:31.065414 31457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:04:31.066152 31492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.066351 31457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.066476 31457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "bbd344e03f4c40f0e9f8f9019d67639c"
server_key_iv: "f66935e35a7b67e47375fc076234193d"
server_key_version: "encryptionkey@0"
I20260927 21:04:31.066852 31457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:31.077944 31457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:31.080406 31457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:31.080575 31457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:31.080868 31457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:31.081334 31499 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:04:31.083324 31457 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:04:31.083395 31457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:04:31.083433 31457 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:04:31.084077 31457 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:04:31.084133 31457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:31.084208 31499 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap starting.
I20260927 21:04:31.091378 31457 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:45097
I20260927 21:04:31.091747 31457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:31.095412 31499 log.cc:824] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:31.096222 31606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:45097 every 8 connection(s)
I20260927 21:04:31.098457 31499 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:04:31.098773 31499 tablet_bootstrap.cc:492] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap complete.
I20260927 21:04:31.099777 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31457
I20260927 21:04:31.100029 31499 ts_tablet_manager.cc:1397] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260927 21:04:31.100098 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:34109
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=38631
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:31.101465 31499 raft_consensus.cc:348] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:31.101752 31499 raft_consensus.cc:729] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a97e87d9aae4f30b09b2eab60fdbd62, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.101917 31499 consensus_queue.cc:260] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } }
I20260927 21:04:31.102175 31499 ts_tablet_manager.cc:1428] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:31.109002 31609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.092498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:51143)
0927 21:04:31.092689 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.092732 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.093283 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.096539 (+ 3256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.096545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.096986 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:31.097195 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.097202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.098597 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.098599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.099655 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.099659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.100774 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.100776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.100991 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.101706 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:31.101719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:31.103782 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:31.106821 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.106828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.106831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.107146 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.107501 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.107502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.107502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.107635 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.108244 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:31.108250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:31.108444 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.108575 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.108763 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260927 21:04:31.109577 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.092582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:51143 (local address 127.16.190.254:36733)
0927 21:04:31.095815 (+ 3233us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.095826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.095918 (+ 92us) server_negotiation.cc:407] Connection header received
0927 21:04:31.096091 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.096102 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.096180 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.096371 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:31.097364 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.098441 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.099772 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.100524 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.100652 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.103962 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:31.103967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:31.103967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:31.103982 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:31.106710 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.107265 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.107270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.107271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.107414 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.107727 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.107729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.107729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.108147 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:31.108262 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.108693 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.108936 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3157,"thread_start_us":107,"threads_started":1}
I20260927 21:04:31.110630 31607 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:31.110749 31607 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:31.110996 31607 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:31.111928 31419 ts_manager.cc:194] Registered new tserver with Master: 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097)
I20260927 21:04:31.113498 31419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:51143
I20260927 21:04:31.114629 31607 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:04:31.123564 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.112704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:45097 (local address 127.0.0.1:56014)
0927 21:04:31.113286 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.113321 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.113386 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.113752 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.113757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.114093 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:31.114154 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.114156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.115844 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.115845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.117006 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.117009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.117648 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.117649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.117734 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.118336 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:31.118342 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:31.120084 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:31.122380 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.122387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.122389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.122590 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.122860 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.122861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.122861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.122929 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.123264 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:31.123270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:31.123316 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.123385 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.123486 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":91,"threads_started":1}
I20260927 21:04:31.123660 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.113103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.16.190.193:45097)
0927 21:04:31.113517 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.113521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.113529 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:04:31.113560 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.113562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.113660 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.114231 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:31.114521 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.115747 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.117107 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.117567 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.117642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.120182 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:31.120192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:31.120192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:31.120201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:31.122276 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.122679 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.122682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.122683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.122754 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.123007 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.123008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.123008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.123191 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:31.123279 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.123452 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.123550 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":89,"threads_started":1}
I20260927 21:04:31.124105 31541 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b093ebf176340d1a59251cddcffe2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:30 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260927 21:04:31.124784 31616 tablet_replica.cc:330] stopping tablet replica
I20260927 21:04:31.124954 31616 raft_consensus.cc:2229] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:31.125043 31616 raft_consensus.cc:2258] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:31.125357 31616 ts_tablet_manager.cc:1905] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:31.127519 31616 ts_tablet_manager.cc:1918] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:31.127647 31616 log.cc:1196] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/8b093ebf176340d1a59251cddcffe2ec
I20260927 21:04:31.127897 31616 ts_tablet_manager.cc:1939] T 8b093ebf176340d1a59251cddcffe2ec P 7a97e87d9aae4f30b09b2eab60fdbd62: Deleting consensus metadata
I20260927 21:04:31.128456 31403 catalog_manager.cc:4841] TS 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097): tablet 8b093ebf176340d1a59251cddcffe2ec (table test-table [id=db58fe0e5bcb48d1a92a29d8ad946f6c]) successfully deleted
W20260927 21:04:31.193994 31612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:31.194216 31612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:31.194242 31612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:31.195461 31612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:31.195523 31612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:31.195550 31612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:04:31.197176 31612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:34109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=38631
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:31.197402 31612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:31.197577 31612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:31.203002 31622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.203027 31624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.203045 31612 server_base.cc:1034] running on GCE node
W20260927 21:04:31.203011 31621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.203467 31612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:31.203670 31612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:31.204859 31612 hybrid_clock.cc:648] HybridClock initialized: now 1790543071204834 us; error 40 us; skew 500 ppm
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:31.207499 31612 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:31.207962 31612 webserver.cc:460] Webserver started at http://127.16.190.194:38631/ using document root <none> and password file <none>
I20260927 21:04:31.208204 31612 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:31.208261 31612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:31.209735 31612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.210494 31631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.210700 31612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.210803 31612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "fffc8e1c33624938a54c0e2a7a581047"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "3e2d9c2487f3c2bd42ddea555aec3aaf"
server_key_iv: "4a35fff433a85b67746ebda58e227d77"
server_key_version: "encryptionkey@0"
I20260927 21:04:31.211180 31612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:31.223798 31612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:31.226217 31612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:31.226507 31612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:31.226773 31612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:31.227115 31612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:31.227173 31612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.227208 31612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:31.227234 31612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.233309 31612 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:34109
I20260927 21:04:31.233359 31744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:34109 every 8 connection(s)
I20260927 21:04:31.233712 31612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:04:31.237035 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31612
I20260927 21:04:31.237347 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:43197
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=43627
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:31.244216 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.234474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:37975 (local address 127.16.190.254:36733)
0927 21:04:31.234563 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.234565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.234758 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:04:31.235267 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.235268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.235281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.235381 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:31.236279 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.236912 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.238490 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.238717 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.238810 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.241141 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:31.241143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:31.241144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:31.241152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:31.242668 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.243177 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.243179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.243179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.243245 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.243513 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.243514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.243514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.243642 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:31.243723 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.244011 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.244120 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 21:04:31.244424 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.234462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:37975)
0927 21:04:31.234621 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.234665 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.235173 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.235411 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.235414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.235994 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:31.236190 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.236195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.237007 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.237008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.238409 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.238413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.238821 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.238829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.238970 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.239551 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:31.239560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:31.241039 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:31.242826 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.242835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.242838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.243089 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.243316 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.243317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.243317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.243442 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.243713 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:31.243719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:31.243874 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.243988 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.244157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 21:04:31.245033 31745 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:31.245144 31745 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:31.245352 31745 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:31.245767 31419 ts_manager.cc:194] Registered new tserver with Master: fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
I20260927 21:04:31.246157 31419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:37975
W20260927 21:04:31.329514 31748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:31.329792 31748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:31.329840 31748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:31.331192 31748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:31.331272 31748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:31.331326 31748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:31.333128 31748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:43197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=43627
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:31.333472 31748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:31.333735 31748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:04:31.339293 31748 server_base.cc:1034] running on GCE node
W20260927 21:04:31.339274 31753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.339311 31754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.339277 31756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.339751 31748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:31.340047 31748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:31.341228 31748 hybrid_clock.cc:648] HybridClock initialized: now 1790543071341215 us; error 44 us; skew 500 ppm
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:31.343878 31748 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:31.344280 31748 webserver.cc:460] Webserver started at http://127.16.190.195:43627/ using document root <none> and password file <none>
I20260927 21:04:31.344527 31748 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:31.344594 31748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:31.346000 31748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.346731 31763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.346935 31748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.347026 31748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3eb90ff82d924946b51e53f002cae203"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "46376d32a8b87f57e00aad85558eaf6c"
server_key_iv: "357624576c00ec9a8c537c5864452faf"
server_key_version: "encryptionkey@0"
I20260927 21:04:31.347386 31748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:31.368508 31748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:31.370991 31748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:31.371143 31748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:31.371392 31748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:31.371719 31748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:31.371753 31748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.371802 31748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:31.371830 31748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.377571 31748 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:43197
I20260927 21:04:31.377612 31876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:43197 every 8 connection(s)
I20260927 21:04:31.377961 31748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:04:31.382885 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31748
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:31.389447 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.378723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:43141 (local address 127.16.190.254:36733)
0927 21:04:31.378800 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.378802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.379026 (+ 224us) server_negotiation.cc:407] Connection header received
0927 21:04:31.379610 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.379612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.379622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.379688 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:31.380727 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.381254 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.382433 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.382681 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.382762 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.385670 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:31.385672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:31.385672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:31.385680 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:31.387533 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.388117 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.388118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.388119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.388185 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.388575 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.388576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.388576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.388722 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:31.388796 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.389269 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.389340 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 21:04:31.389699 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.378698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:43141)
0927 21:04:31.378876 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.378919 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.379480 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.379770 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.379773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.380386 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:31.380592 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.380596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.381433 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.381434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.382297 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.382299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.382779 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.382781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.382916 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.383787 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:31.383798 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:31.385568 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:31.387658 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.387665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.387667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.388017 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.388273 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:31.388274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:31.388274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:31.388475 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:31.388881 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:31.388888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:31.389077 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.389225 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.389421 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260927 21:04:31.390242 31877 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36733
I20260927 21:04:31.390420 31877 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:31.390631 31877 heartbeater.cc:511] Master 127.16.190.254:36733 requested a full tablet report, sending...
I20260927 21:04:31.391008 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.383370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.16.190.254:36733)
0927 21:04:31.383748 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.383751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.383760 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:31.383824 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.383825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.383839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.383921 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:31.384494 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.385061 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.385966 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.386283 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.386343 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.387818 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:31.387819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:31.387819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:31.387827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:31.389623 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.390190 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.390191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.390191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.390256 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:31.390478 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:31.390479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:31.390479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:31.390587 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:31.390766 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.390802 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.390894 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":66,"threads_started":1}
I20260927 21:04:31.391119 31419 ts_manager.cc:194] Registered new tserver with Master: 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197)
I20260927 21:04:31.391580 31419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:43141
I20260927 21:04:31.391744 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:04:31.394582 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.392101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.16.190.254:36733)
0927 21:04:31.392141 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.392143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.392261 (+ 118us) server_negotiation.cc:407] Connection header received
0927 21:04:31.392334 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.392335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.392362 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.392419 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:04:31.392550 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.393072 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.393857 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.394065 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.394127 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.394306 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:04:31.394411 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:04:31.394476 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.394510 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.394531 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:04:31.394882 31419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:04:31.396509 31419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:04:31.399025 31541 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 (DEFAULT_TABLE table=test-table [id=33b0615151f74792a4b71e7cbf55302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:31.399288 31541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb52425dc31493db2a28a6c315cbfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:31.400717 31499 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap starting.
I20260927 21:04:31.401099 31499 tablet_bootstrap.cc:654] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:31.402113 31499 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: No bootstrap required, opened a new log
I20260927 21:04:31.402176 31499 ts_tablet_manager.cc:1397] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.402328 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.398624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34109 (local address 127.0.0.1:45094)
0927 21:04:31.398736 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.398750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.398790 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.399138 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.399139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.399140 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:31.399188 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.399190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.400237 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.400238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.401190 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.401192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.402049 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.402051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.402125 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.402149 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.402209 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.402264 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 21:04:31.402328 31499 raft_consensus.cc:348] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.402407 31499 raft_consensus.cc:374] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:31.402462 31499 raft_consensus.cc:729] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a97e87d9aae4f30b09b2eab60fdbd62, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.402508 31499 consensus_queue.cc:260] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.402567 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.398800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.16.190.194:34109)
0927 21:04:31.398974 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.398977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.398985 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:04:31.399019 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.399021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.399034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.399134 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:31.399244 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.400143 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.401253 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.401968 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.402044 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.402096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.402430 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.402485 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260927 21:04:31.402617 31499 ts_tablet_manager.cc:1428] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:04:31.403432 31679 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 (DEFAULT_TABLE table=test-table [id=33b0615151f74792a4b71e7cbf55302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:31.403807 31679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb52425dc31493db2a28a6c315cbfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:31.404914 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.398624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43197 (local address 127.0.0.1:58006)
0927 21:04:31.400979 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.400991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.401052 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.401388 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.401390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.401392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:31.401447 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.401449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.402790 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.402791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.403836 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.403839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.404645 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.404646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.404705 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.404710 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.404768 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.404839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2331,"mutex_wait_us":12,"thread_start_us":99,"threads_started":1}
I20260927 21:04:31.404928 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.398713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.16.190.195:43197)
0927 21:04:31.398875 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.398880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.401086 (+ 2206us) server_negotiation.cc:407] Connection header received
0927 21:04:31.401143 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.401146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.401163 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.401283 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:31.401546 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.402618 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.403964 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.404556 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.404648 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.404701 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.404783 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.404847 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260927 21:04:31.405642 31811 tablet_service.cc:1467] Processing CreateTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 (DEFAULT_TABLE table=test-table [id=33b0615151f74792a4b71e7cbf55302f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:31.405980 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fb52425dc31493db2a28a6c315cbfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:31.407317 31887 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Bootstrap starting.
I20260927 21:04:31.407913 31887 tablet_bootstrap.cc:654] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:31.408434 31887 log.cc:824] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:31.409210 31889 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Bootstrap starting.
I20260927 21:04:31.409240 31887 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: No bootstrap required, opened a new log
I20260927 21:04:31.409312 31887 ts_tablet_manager.cc:1397] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:04:31.409767 31889 tablet_bootstrap.cc:654] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:31.410260 31889 log.cc:824] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:31.410975 31889 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: No bootstrap required, opened a new log
I20260927 21:04:31.410923 31887 raft_consensus.cc:348] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.411062 31889 ts_tablet_manager.cc:1397] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:04:31.411077 31887 raft_consensus.cc:374] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:31.411121 31887 raft_consensus.cc:729] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc8e1c33624938a54c0e2a7a581047, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.411252 31887 consensus_queue.cc:260] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.411546 31887 ts_tablet_manager.cc:1428] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:04:31.411584 31745 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
I20260927 21:04:31.413002 31889 raft_consensus.cc:348] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.413149 31889 raft_consensus.cc:374] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:31.413194 31889 raft_consensus.cc:729] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb90ff82d924946b51e53f002cae203, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.413333 31889 consensus_queue.cc:260] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.413609 31889 ts_tablet_manager.cc:1428] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:04:31.413667 31877 heartbeater.cc:503] Master 127.16.190.254:36733 was elected leader, sending a full tablet report...
W20260927 21:04:31.484295 31746 tablet.cc:2369] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:31.594111 31891 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:31.594249 31891 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.594590 31891 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097)
W20260927 21:04:31.596961 31608 tablet.cc:2369] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:31.598081 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.594947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:41579 (local address 127.16.190.195:43197)
0927 21:04:31.595011 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.595013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.595022 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:31.595103 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.595105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.595116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.595204 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:31.595463 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.595989 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597289 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.597684 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597744 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.597750 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.597960 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.598009 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:04:31.598105 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.594931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:58989 (local address 127.16.190.193:45097)
0927 21:04:31.594982 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.594984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.595249 (+ 265us) server_negotiation.cc:407] Connection header received
0927 21:04:31.595291 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.595293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.595323 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.595407 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:31.595568 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.596165 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597291 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.597767 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.597931 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.597997 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.598037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:04:31.598105 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.594868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43197 (local address 127.16.190.194:41579)
0927 21:04:31.594940 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.594954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.595007 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.595250 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.595251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.595253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:31.595334 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.595336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.596084 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.596086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.597192 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.597195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.597789 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.597881 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.597886 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.597945 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.598003 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":6}
I20260927 21:04:31.598116 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.594868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:45097 (local address 127.16.190.194:58989)
0927 21:04:31.595163 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.595174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.595235 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:31.595415 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:31.595416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:31.595417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:31.595493 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.595495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.596255 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.596256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.597173 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:31.597176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.597851 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.597852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:31.597912 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.597916 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:04:31.597993 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:04:31.598028 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":106,"thread_start_us":111,"threads_started":1}
I20260927 21:04:31.598529 31831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb52425dc31493db2a28a6c315cbfa3" candidate_uuid: "fffc8e1c33624938a54c0e2a7a581047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb90ff82d924946b51e53f002cae203" is_pre_election: true
I20260927 21:04:31.598697 31831 raft_consensus.cc:2454] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fffc8e1c33624938a54c0e2a7a581047 in term 0.
I20260927 21:04:31.598658 31561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb52425dc31493db2a28a6c315cbfa3" candidate_uuid: "fffc8e1c33624938a54c0e2a7a581047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" is_pre_election: true
I20260927 21:04:31.598861 31561 raft_consensus.cc:2454] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fffc8e1c33624938a54c0e2a7a581047 in term 0.
I20260927 21:04:31.598948 31634 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047; no voters:
I20260927 21:04:31.599079 31891 raft_consensus.cc:2781] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:04:31.599126 31891 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:04:31.599153 31891 raft_consensus.cc:3037] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:31.599874 31891 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.600034 31891 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097)
I20260927 21:04:31.600167 31831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb52425dc31493db2a28a6c315cbfa3" candidate_uuid: "fffc8e1c33624938a54c0e2a7a581047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb90ff82d924946b51e53f002cae203"
I20260927 21:04:31.600243 31561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fb52425dc31493db2a28a6c315cbfa3" candidate_uuid: "fffc8e1c33624938a54c0e2a7a581047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
I20260927 21:04:31.600250 31831 raft_consensus.cc:3037] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:31.600317 31561 raft_consensus.cc:3037] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:31.601178 31561 raft_consensus.cc:2454] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fffc8e1c33624938a54c0e2a7a581047 in term 1.
I20260927 21:04:31.601178 31831 raft_consensus.cc:2454] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fffc8e1c33624938a54c0e2a7a581047 in term 1.
I20260927 21:04:31.601349 31634 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047; no voters:
I20260927 21:04:31.601468 31891 raft_consensus.cc:2781] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:31.601624 31891 raft_consensus.cc:686] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 1 LEADER]: Becoming Leader. State: Replica: fffc8e1c33624938a54c0e2a7a581047, State: Running, Role: LEADER
I20260927 21:04:31.601730 31891 consensus_queue.cc:237] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.602666 31419 catalog_manager.cc:5495] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 reported cstate change: term changed from 0 to 1, leader changed from <none> to fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "fffc8e1c33624938a54c0e2a7a581047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } health_report { overall_health: UNKNOWN } } }
W20260927 21:04:31.628683 31878 tablet.cc:2369] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:31.631575 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.629083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.16.190.194:34109)
0927 21:04:31.629177 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:31.629179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:31.629189 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:04:31.629255 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:31.629256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:31.629277 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:31.629367 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:04:31.629482 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.629967 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.630740 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:31.630954 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:31.631044 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:31.631227 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:04:31.631336 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:04:31.631431 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:31.631453 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 21:04:31.631478 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 21:04:31.633569 31831 raft_consensus.cc:1261] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Refusing update from remote peer fffc8e1c33624938a54c0e2a7a581047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:31.633569 31561 raft_consensus.cc:1261] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Refusing update from remote peer fffc8e1c33624938a54c0e2a7a581047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:31.633944 31894 consensus_queue.cc:1035] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:31.634152 31891 consensus_queue.cc:1035] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:31.636550 31897 mvcc.cc:205] Tried to move back new op lower bound from 7334064421407657984 to 7334064421281239040. Current Snapshot: MvccSnapshot[applied={T|T < 7334064421407657984}]
I20260927 21:04:31.636739 31900 mvcc.cc:205] Tried to move back new op lower bound from 7334064421407657984 to 7334064421281239040. Current Snapshot: MvccSnapshot[applied={T|T < 7334064421407657984}]
I20260927 21:04:31.639037 31419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:31.639143 31419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:31.639983 31419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Sending DeleteTablet for 3 replicas of tablet 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:31.640388 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:31 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:58006
I20260927 21:04:31.640403 31679 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:31 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:45094
I20260927 21:04:31.640471 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31457
I20260927 21:04:31.640599 31541 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fb52425dc31493db2a28a6c315cbfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:31 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56014
I20260927 21:04:31.641186 31908 tablet_replica.cc:330] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: stopping tablet replica
I20260927 21:04:31.641286 31908 raft_consensus.cc:2229] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:04:31.641299 31907 tablet_replica.cc:330] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: stopping tablet replica
I20260927 21:04:31.641404 31907 raft_consensus.cc:2229] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:31.641414 31908 raft_consensus.cc:2258] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:31.641511 31907 raft_consensus.cc:2258] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:31.641985 31907 ts_tablet_manager.cc:1905] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:31.642004 31908 ts_tablet_manager.cc:1905] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:31.643635 31907 ts_tablet_manager.cc:1918] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:31.643695 31908 ts_tablet_manager.cc:1918] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:31.643764 31908 log.cc:1196] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:31.643769 31907 log.cc:1196] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:31.644107 31907 ts_tablet_manager.cc:1939] T 8fb52425dc31493db2a28a6c315cbfa3 P 3eb90ff82d924946b51e53f002cae203: Deleting consensus metadata
I20260927 21:04:31.644230 31908 ts_tablet_manager.cc:1939] T 8fb52425dc31493db2a28a6c315cbfa3 P fffc8e1c33624938a54c0e2a7a581047: Deleting consensus metadata
I20260927 21:04:31.644659 31405 catalog_manager.cc:4841] TS 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): tablet 8fb52425dc31493db2a28a6c315cbfa3 (table test-table [id=33b0615151f74792a4b71e7cbf55302f]) successfully deleted
I20260927 21:04:31.645390 31404 catalog_manager.cc:4841] TS fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): tablet 8fb52425dc31493db2a28a6c315cbfa3 (table test-table [id=33b0615151f74792a4b71e7cbf55302f]) successfully deleted
W20260927 21:04:31.647256 31403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:45097 (error 108)
I20260927 21:04:31.647809 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:31.647663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:45097 (local address 127.0.0.1:56026)
0927 21:04:31.647723 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.647763 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:04:31.647964 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31612
W20260927 21:04:31.649077 31403 catalog_manager.cc:4584] TS 7a97e87d9aae4f30b09b2eab60fdbd62 (127.16.190.193:45097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fb52425dc31493db2a28a6c315cbfa3: Network error: Client connection negotiation failed: client connection to 127.16.190.193:45097: connect: Connection refused (error 111)
I20260927 21:04:31.653468 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31748
I20260927 21:04:31.658622 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31386
I20260927 21:04:31.663926 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--webserver_interface=127.16.190.254
--webserver_port=46149
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:31.745085 31910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:31.745302 31910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:31.745325 31910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:31.746505 31910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:31.746554 31910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:31.746569 31910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:31.746582 31910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:31.746595 31910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:31.748145 31910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=46149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:31.748404 31910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:31.748636 31910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:31.753599 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.753589 31915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.753635 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.753623 31910 server_base.cc:1034] running on GCE node
I20260927 21:04:31.754050 31910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:31.754354 31910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:31.755548 31910 hybrid_clock.cc:648] HybridClock initialized: now 1790543071755529 us; error 27 us; skew 500 ppm
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:31.758253 31910 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:31.758778 31910 webserver.cc:460] Webserver started at http://127.16.190.254:46149/ using document root <none> and password file <none>
I20260927 21:04:31.759016 31910 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:31.759084 31910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:31.760541 31910 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:04:31.761269 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.761436 31910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.761540 31910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a648c63003a648cf9a5a1c95551b1f98"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "6c846a9ba47b497712c1fdf91f85ee09"
server_key_iv: "bb9391824c285395b5f522e91c2306f6"
server_key_version: "encryptionkey@0"
I20260927 21:04:31.761907 31910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:31.774612 31910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:31.777346 31910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:31.777540 31910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:31.781960 31910 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36733
I20260927 21:04:31.781993 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36733 every 8 connection(s)
I20260927 21:04:31.782413 31910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:31.784171 31978 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:04:31.784777 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap starting.
I20260927 21:04:31.789109 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31910
I20260927 21:04:31.789410 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:45097
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=41325
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:31.794586 31978 log.cc:824] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:31.797050 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:04:31.797416 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98: Bootstrap complete.
I20260927 21:04:31.799751 31978 raft_consensus.cc:348] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:31.799990 31978 raft_consensus.cc:729] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.800174 31978 consensus_queue.cc:260] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:31.800254 31978 raft_consensus.cc:388] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:31.800307 31978 raft_consensus.cc:482] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:31.800359 31978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:04:31.801544 31978 raft_consensus.cc:504] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:31.801664 31978 leader_election.cc:302] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a648c63003a648cf9a5a1c95551b1f98; no voters:
I20260927 21:04:31.801878 31978 leader_election.cc:288] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:04:31.801926 31982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:04:31.802047 31982 raft_consensus.cc:686] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [term 3 LEADER]: Becoming Leader. State: Replica: a648c63003a648cf9a5a1c95551b1f98, State: Running, Role: LEADER
I20260927 21:04:31.802148 31982 consensus_queue.cc:237] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } }
I20260927 21:04:31.802175 31978 sys_catalog.cc:564] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:31.802448 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a648c63003a648cf9a5a1c95551b1f98. Latest consensus state: current_term: 3 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:31.802534 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:31.802453 31984 sys_catalog.cc:455] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a648c63003a648cf9a5a1c95551b1f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a648c63003a648cf9a5a1c95551b1f98" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36733 } } }
I20260927 21:04:31.802721 31984 sys_catalog.cc:458] T 00000000000000000000000000000000 P a648c63003a648cf9a5a1c95551b1f98 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:31.803123 31986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:31.804045 31986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:31.804176 31986 catalog_manager.cc:1259] Loaded cluster ID: 9ba86eae9aff4205a978d543325e9abd
I20260927 21:04:31.804207 31986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 21:04:31.804598 31986 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcd0cadc9ad google::LogMessage::Fail() at ??:0
@ 0x7fcd0cadf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcd0cadc4de google::LogMessage::Flush() at ??:0
@ 0x7fcd0cadda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcd149fd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcd14a068b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcd0e9aa0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcd0e99f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcd0f8036db start_thread at ??:0
@ 0x7fcd0b5da61f clone at ??:0
W20260927 21:04:31.886096 31980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:31.886350 31980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:31.886391 31980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:31.887591 31980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:31.887661 31980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:31.887705 31980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:31.889302 31980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:45097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=41325
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:31.889588 31980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:31.889829 31980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:31.896087 32015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:31.896116 32017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.896363 31980 server_base.cc:1034] running on GCE node
W20260927 21:04:31.896526 32014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:31.896776 31980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:31.897114 31980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:31.898303 31980 hybrid_clock.cc:648] HybridClock initialized: now 1790543071898286 us; error 29 us; skew 500 ppm
Sep 27 21:04:31 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:31.901450 31980 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:31.901993 31980 webserver.cc:460] Webserver started at http://127.16.190.193:41325/ using document root <none> and password file <none>
I20260927 21:04:31.902271 31980 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:31.902398 31980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:31.903965 31980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.904742 32024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:31.904960 31980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:31.905083 31980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "7a97e87d9aae4f30b09b2eab60fdbd62"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "bbd344e03f4c40f0e9f8f9019d67639c"
server_key_iv: "f66935e35a7b67e47375fc076234193d"
server_key_version: "encryptionkey@0"
I20260927 21:04:31.905467 31980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:31.921965 31980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:31.924424 31980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:31.924609 31980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:31.924916 31980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:31.925434 32031 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:04:31.927615 31980 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:04:31.927697 31980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:04:31.927747 31980 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:04:31.928447 31980 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:04:31.928504 31980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:31.928583 32031 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap starting.
I20260927 21:04:31.936106 31980 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:45097
I20260927 21:04:31.936529 31980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:31.940184 32031 log.cc:824] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:31.940294 32138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:45097 every 8 connection(s)
I20260927 21:04:31.943150 32031 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:04:31.943480 32031 tablet_bootstrap.cc:492] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Bootstrap complete.
I20260927 21:04:31.944728 32031 ts_tablet_manager.cc:1397] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260927 21:04:31.945416 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 31980
I20260927 21:04:31.945688 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:34109
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=38631
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:31.946305 32031 raft_consensus.cc:348] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.946678 32031 raft_consensus.cc:729] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a97e87d9aae4f30b09b2eab60fdbd62, State: Initialized, Role: FOLLOWER
I20260927 21:04:31.946839 32031 consensus_queue.cc:260] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:31.947144 32031 ts_tablet_manager.cc:1428] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260927 21:04:32.038033 32143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:32.038252 32143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:32.038277 32143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:32.039577 32143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:32.039628 32141 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:04:31.937255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:43067)
0927 21:04:31.937454 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:31.937513 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:31.937993 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:32.039374 (+101381us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
W20260927 21:04:32.039654 32143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:32.039705 32143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
W20260927 21:04:32.039945 32139 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36733 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36733: Client connection negotiation failed: client connection to 127.16.190.254:36733: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:04:32.040347 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.040191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:34675)
0927 21:04:32.040249 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.040296 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:04:32.041181 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.041066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:48699)
0927 21:04:32.041117 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.041146 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:04:32.041307 32139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:04:32.041666 32143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:34109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=38631
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:32.041954 32143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:32.042198 32143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:32.047919 32149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:32.047897 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:32.047919 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:32.048199 32143 server_base.cc:1034] running on GCE node
I20260927 21:04:32.048382 32143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:32.048617 32143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:32.049778 32143 hybrid_clock.cc:648] HybridClock initialized: now 1790543072049753 us; error 45 us; skew 500 ppm
Sep 27 21:04:32 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543072, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:32.052717 32143 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:32.053256 32143 webserver.cc:460] Webserver started at http://127.16.190.194:38631/ using document root <none> and password file <none>
I20260927 21:04:32.053471 32143 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:32.053515 32143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:32.054816 32143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:32.055394 32159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.055590 32143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:32.055802 32143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "fffc8e1c33624938a54c0e2a7a581047"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "3e2d9c2487f3c2bd42ddea555aec3aaf"
server_key_iv: "4a35fff433a85b67746ebda58e227d77"
server_key_version: "encryptionkey@0"
I20260927 21:04:32.056152 32143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:32.064926 32143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:32.067121 32143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:32.067239 32143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:32.067430 32143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:32.067755 32143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:32.067791 32143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.067814 32143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:32.067827 32143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.074029 32143 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:34109
I20260927 21:04:32.074072 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:34109 every 8 connection(s)
I20260927 21:04:32.074452 32143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:04:32.075050 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 32143
I20260927 21:04:32.075364 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:43197
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:36733
--webserver_port=43627
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:04:32.075897 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.075231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:40653)
0927 21:04:32.075590 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.075728 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":85,"threads_started":1}
W20260927 21:04:32.076475 32273 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36733 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36733: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
I20260927 21:04:32.076896 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.076718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:33009)
0927 21:04:32.076790 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.076857 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 21:04:32.077278 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.077163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:48709)
0927 21:04:32.077217 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.077252 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 21:04:32.077605 32273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:04:32.158463 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:32.158782 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:32.158823 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:32.160084 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:32.160152 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:32.160199 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:32.161803 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:43197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=43627
--tserver_master_addrs=127.16.190.254:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:32.162092 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:32.162334 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:32.167622 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:32.167747 32276 server_base.cc:1034] running on GCE node
W20260927 21:04:32.167650 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:32.167729 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:32.168128 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:32.168386 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:32.169597 32276 hybrid_clock.cc:648] HybridClock initialized: now 1790543072169585 us; error 37 us; skew 500 ppm
Sep 27 21:04:32 dist-test-slave-6c7s krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543072, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:32.172349 32276 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:32.172789 32276 webserver.cc:460] Webserver started at http://127.16.190.195:43627/ using document root <none> and password file <none>
I20260927 21:04:32.173112 32276 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:32.173163 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:32.174618 32276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:32.175316 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.175549 32276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:32.175643 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "3eb90ff82d924946b51e53f002cae203"
format_stamp: "Formatted at 2026-09-27 21:04:30 on dist-test-slave-6c7s"
server_key: "46376d32a8b87f57e00aad85558eaf6c"
server_key_iv: "357624576c00ec9a8c537c5864452faf"
server_key_version: "encryptionkey@0"
I20260927 21:04:32.176038 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:32.238778 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:32.241523 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:32.241719 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:32.241977 32276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:32.242319 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:32.242378 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.242421 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:32.242451 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:32.248703 32276 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:43197
I20260927 21:04:32.248747 32404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:43197 every 8 connection(s)
I20260927 21:04:32.249195 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:04:32.250360 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.249942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:50325)
0927 21:04:32.250119 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.250210 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
W20260927 21:04:32.250802 32405 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36733 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36733: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
I20260927 21:04:32.251157 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.251031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:36921)
0927 21:04:32.251101 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.251134 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 21:04:32.251580 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:32.251480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:46945)
0927 21:04:32.251533 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:32.251561 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 21:04:32.251731 32405 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:04:32.252292 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 32276
I20260927 21:04:33.042011 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.041744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:50443)
0927 21:04:33.041890 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:33.041932 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260927 21:04:33.078352 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.078083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:48181)
0927 21:04:33.078243 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:33.078288 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":59,"threads_started":1}
I20260927 21:04:33.252414 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.252138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:50791)
0927 21:04:33.252286 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:33.252353 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260927 21:04:33.511670 32412 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:33.511795 32412 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:33.512085 32412 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
Sep 27 21:04:33 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.194@KRBTEST.COM
Sep 27 21:04:33 dist-test-slave-6c7s krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543071, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.195@KRBTEST.COM
I20260927 21:04:33.523351 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.512305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:43197 (local address 127.16.190.193:51899)
0927 21:04:33.512364 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:33.512376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:33.512435 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:33.513520 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:33.513523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:33.513940 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:33.514057 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:33.514063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.515179 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.515180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:33.515976 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:33.515978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.516584 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:33.516699 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:33.517430 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:33.517434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:33.519394 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:33.521459 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:33.521459 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:33.521460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:33.521715 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:33.522025 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:33.522026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:33.522026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:33.522232 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:33.522737 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:33.522743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:33.522908 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:04:33.523043 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 21:04:33.523233 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":32}
I20260927 21:04:33.523372 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.512425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:51899 (local address 127.16.190.195:43197)
0927 21:04:33.512589 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:33.512602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:33.512636 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:04:33.513253 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:33.513256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:33.513316 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:33.513478 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:33.514207 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.515069 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516071 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.516478 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:33.519507 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:33.519511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:33.519511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:33.519523 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:33.521383 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:33.521844 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:33.521847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:33.521847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:33.521943 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:33.522321 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:33.522322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:33.522323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:33.522656 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:33.522747 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:33.523118 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 21:04:33.523254 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260927 21:04:33.523432 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.512460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:43515 (local address 127.16.190.194:34109)
0927 21:04:33.512598 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:33.512605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:33.512636 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:04:33.513254 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:33.513257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:33.513316 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:33.513479 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:33.514185 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.515070 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516078 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.516511 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516645 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:33.519124 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:33.519129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:33.519129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:33.519143 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:33.521214 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:33.521817 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:33.521821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:33.521821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:33.521943 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:33.522384 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:33.522385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:33.522385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:33.522656 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:33.522747 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:33.523113 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:04:33.523254 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260927 21:04:33.523372 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:33.512305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34109 (local address 127.16.190.193:43515)
0927 21:04:33.512522 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:33.512530 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:33.512567 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:33.513628 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:33.513630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:33.513857 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:33.514056 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:33.514062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.515175 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.515176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:33.515976 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:33.515978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:33.516609 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:33.516611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:33.516700 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:33.517353 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:33.517362 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:33.518950 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:33.521299 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:33.521306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:33.521308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:33.521716 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:33.522023 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:33.522023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:33.522024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:33.522232 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:33.522737 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:33.522743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:33.522908 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 21:04:33.523026 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 21:04:33.523287 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":258,"thread_start_us":74,"threads_started":1}
W20260927 21:04:33.524871 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:33.524971 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:33.525022 32026 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:33.525130 32412 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:34.042862 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:34.042578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:51613)
0927 21:04:34.042766 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:34.042813 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260927 21:04:34.079197 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:34.078960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:43831)
0927 21:04:34.079105 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:34.079152 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260927 21:04:34.253175 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:34.252983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:39539)
0927 21:04:34.253097 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:34.253132 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260927 21:04:35.043699 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:35.043454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:54697)
0927 21:04:35.043604 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:35.043653 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260927 21:04:35.080013 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:35.079783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:42077)
0927 21:04:35.079924 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:35.079964 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260927 21:04:35.253912 32421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:35.253664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:45597)
0927 21:04:35.253831 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:35.253867 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260927 21:04:35.467527 32422 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:35.467636 32422 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:35.467828 32422 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
W20260927 21:04:35.468355 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:35.468559 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:35.468631 32026 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:35.468765 32422 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:36.044556 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:36.044264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:48957)
0927 21:04:36.044463 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:36.044509 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260927 21:04:36.080935 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:36.080623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:36905)
0927 21:04:36.080788 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:36.080886 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
I20260927 21:04:36.254767 32425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:36.254552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:33983)
0927 21:04:36.254691 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:36.254725 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260927 21:04:37.045501 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:37.045199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:36731)
0927 21:04:37.045396 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:37.045451 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260927 21:04:37.081754 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:37.081469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:60643)
0927 21:04:37.081626 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:37.081687 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260927 21:04:37.255548 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:37.255269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:33719)
0927 21:04:37.255450 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:37.255501 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":55,"threads_started":1}
I20260927 21:04:38.046217 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:38.046013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:35571)
0927 21:04:38.046126 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:38.046161 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260927 21:04:38.082543 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:38.082290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:59995)
0927 21:04:38.082452 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:38.082492 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260927 21:04:38.256430 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:38.256143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:43243)
0927 21:04:38.256310 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:38.256367 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260927 21:04:38.669380 32432 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:38.669498 32432 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:38.669677 32432 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
W20260927 21:04:38.670053 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:38.670159 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:38.670246 32026 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:38.670322 32432 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:39.046981 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:39.046719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:51627)
0927 21:04:39.046865 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:39.046915 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260927 21:04:39.083235 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:39.083022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:34469)
0927 21:04:39.083152 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:39.083190 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260927 21:04:39.257211 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:39.256975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:54641)
0927 21:04:39.257118 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:39.257162 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260927 21:04:40.047848 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:40.047553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:58383)
0927 21:04:40.047715 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:40.047767 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260927 21:04:40.083990 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:40.083739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:45513)
0927 21:04:40.083884 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:40.083929 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260927 21:04:40.257936 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:40.257714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:37853)
0927 21:04:40.257850 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:40.257887 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260927 21:04:40.410962 32439 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:40.411075 32439 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:40.411258 32439 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
W20260927 21:04:40.411648 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:40.411728 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:40.411800 32027 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:40.411888 32439 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:41.048691 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:41.048389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:51545)
0927 21:04:41.048591 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:41.048638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260927 21:04:41.084707 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:41.084476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:59013)
0927 21:04:41.084624 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:41.084658 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260927 21:04:41.258734 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:41.258484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:32901)
0927 21:04:41.258642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:41.258687 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260927 21:04:42.049531 32443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:42.049260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:33599)
0927 21:04:42.049415 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:42.049481 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260927 21:04:42.085443 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:42.085198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:35485)
0927 21:04:42.085321 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:42.085362 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260927 21:04:42.259550 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:42.259282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:52463)
0927 21:04:42.259462 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:42.259501 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":125,"threads_started":1}
I20260927 21:04:43.050297 32446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:43.050052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:55513)
0927 21:04:43.050200 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:43.050249 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260927 21:04:43.086304 32447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:43.086089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:44355)
0927 21:04:43.086222 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:43.086256 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 21:04:43.260371 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:43.260136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:50025)
0927 21:04:43.260266 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:43.260313 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260927 21:04:43.550732 32449 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:43.550884 32449 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:43.551110 32449 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
W20260927 21:04:43.551648 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:43.551754 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:43.551836 32026 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:43.551908 32449 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:44.051159 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:44.050876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:60983)
0927 21:04:44.051056 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:44.051102 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260927 21:04:44.087066 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:44.086827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:52433)
0927 21:04:44.086970 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:44.087017 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260927 21:04:44.261209 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:44.260994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:54125)
0927 21:04:44.261116 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:44.261160 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260927 21:04:45.052073 32453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:45.051733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:60631)
0927 21:04:45.051963 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:45.052013 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":68,"threads_started":1}
I20260927 21:04:45.087715 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:45.087504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:56157)
0927 21:04:45.087627 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:45.087666 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260927 21:04:45.261999 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:45.261753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:48421)
0927 21:04:45.261907 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:45.261951 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 21:04:46.052932 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:46.052605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:50453)
0927 21:04:46.052775 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:46.052882 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260927 21:04:46.088517 32457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:46.088243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:51917)
0927 21:04:46.088424 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:46.088468 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260927 21:04:46.262768 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:46.262554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:48301)
0927 21:04:46.262680 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:46.262718 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260927 21:04:46.686208 32459 raft_consensus.cc:482] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:46.686303 32459 raft_consensus.cc:504] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb90ff82d924946b51e53f002cae203" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 43197 } } peers { permanent_uuid: "fffc8e1c33624938a54c0e2a7a581047" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34109 } } peers { permanent_uuid: "7a97e87d9aae4f30b09b2eab60fdbd62" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 45097 } }
I20260927 21:04:46.686518 32459 leader_election.cc:288] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197), fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109)
W20260927 21:04:46.686928 32026 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fffc8e1c33624938a54c0e2a7a581047 (127.16.190.194:34109): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
W20260927 21:04:46.687036 32027 leader_election.cc:341] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3eb90ff82d924946b51e53f002cae203 (127.16.190.195:43197): Not found: Tablet not found: 8fb52425dc31493db2a28a6c315cbfa3
I20260927 21:04:46.687075 32027 leader_election.cc:302] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a97e87d9aae4f30b09b2eab60fdbd62; no voters: 3eb90ff82d924946b51e53f002cae203, fffc8e1c33624938a54c0e2a7a581047
I20260927 21:04:46.687161 32459 raft_consensus.cc:2726] T 8fb52425dc31493db2a28a6c315cbfa3 P 7a97e87d9aae4f30b09b2eab60fdbd62 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:04:47.053953 32460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:47.053710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.193:33393)
0927 21:04:47.053848 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:47.053894 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260927 21:04:47.089300 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:47.089094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.194:49223)
0927 21:04:47.089209 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:47.089238 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260927 21:04:47.256667 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 31980
I20260927 21:04:47.261816 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 32143
I20260927 21:04:47.263583 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:47.263359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36733 (local address 127.16.190.195:56091)
0927 21:04:47.263503 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:47.263540 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
I20260927 21:04:47.266422 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 32276
2026-09-27T21:04:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18488 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:04:47 dist-test-slave-6c7s krb5kdc[32465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:04:47 dist-test-slave-6c7s krb5kdc[32465](info): set up 2 sockets
Sep 27 21:04:47 dist-test-slave-6c7s krb5kdc[32465](info): commencing operation
krb5kdc: starting...
W20260927 21:04:48.384459 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:04:48.517386 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:48.600713 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:48.600978 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:48.601002 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:48.602301 32478 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:04:48.602352 32478 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:04:48.602366 32478 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:04:48.602377 32478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:04:48.602517 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:48.602530 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:48.602543 32478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:04:48.602555 32478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:04:48.604043 32478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:48.604250 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:48.604436 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:04:48.609820 32478 server_base.cc:1034] running on GCE node
W20260927 21:04:48.609787 32484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.609781 32483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.609787 32486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:48.610335 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:48.610654 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:48.611822 32478 hybrid_clock.cc:648] HybridClock initialized: now 1790543088611809 us; error 30 us; skew 500 ppm
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:48.614740 32478 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:04:48.615237 32478 webserver.cc:460] Webserver started at http://127.16.190.254:45365/ using document root <none> and password file <none>
I20260927 21:04:48.615499 32478 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:48.615569 32478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:48.615715 32478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:48.616897 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "a6229a26c9914480bc03c6a5d0a6dec9"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cbe1ad79ab065ab6971dae9c2d1498ac"
server_key_iv: "ddaa78d6d6cdcecb7aeb1276c2901baf"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.617306 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "a6229a26c9914480bc03c6a5d0a6dec9"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cbe1ad79ab065ab6971dae9c2d1498ac"
server_key_iv: "ddaa78d6d6cdcecb7aeb1276c2901baf"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.618778 32478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:04:48.619589 32493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:48.619809 32478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:04:48.619923 32478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a6229a26c9914480bc03c6a5d0a6dec9"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cbe1ad79ab065ab6971dae9c2d1498ac"
server_key_iv: "ddaa78d6d6cdcecb7aeb1276c2901baf"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.620002 32478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:48.636193 32478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:48.638804 32478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:48.638993 32478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:48.643246 32478 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36853
I20260927 21:04:48.643280 32545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36853 every 8 connection(s)
I20260927 21:04:48.643690 32478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:04:48.644613 32546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:48.647027 32546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap starting.
I20260927 21:04:48.647650 32546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:48.647959 32546 log.cc:824] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:48.648770 32546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: No bootstrap required, opened a new log
I20260927 21:04:48.650439 32546 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:04:48.650583 32546 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:48.650606 32546 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Initialized, Role: FOLLOWER
I20260927 21:04:48.650746 32546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:04:48.650835 32546 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:04:48.650882 32546 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:04:48.650913 32546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:48.651443 32546 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:04:48.651532 32546 leader_election.cc:302] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6229a26c9914480bc03c6a5d0a6dec9; no voters:
I20260927 21:04:48.651664 32546 leader_election.cc:288] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:04:48.651764 32549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:48.651927 32546 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:04:48.651932 32549 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 LEADER]: Becoming Leader. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Running, Role: LEADER
I20260927 21:04:48.652153 32549 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:04:48.652683 32551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6229a26c9914480bc03c6a5d0a6dec9. Latest consensus state: current_term: 1 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:04:48.652741 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 32478
I20260927 21:04:48.652769 32551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:48.652889 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:04:48.653046 32550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:04:48.653110 17147 external_mini_cluster.cc:1428] Setting key e1cb8753812c709cbd3784b6073eb286
I20260927 21:04:48.653196 32550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:04:48.653307 32558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:04:48.653843 32558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:04:48.655550 32558 catalog_manager.cc:1347] Generated new cluster ID: b77672ebc59743ae92aeb3bfeb774ac3
I20260927 21:04:48.655599 32558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:48.665537 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:48.653765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.16.190.254:36853)
0927 21:04:48.653938 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:48.653949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:48.653985 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:04:48.654116 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:48.654124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:48.654183 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:48.654328 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:48.655026 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.656407 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.657534 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.658180 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.658303 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:48.661444 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:48.661449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:48.661450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:48.661463 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:48.663558 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:48.664068 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:48.664071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:48.664072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:48.664189 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:48.664489 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:48.664490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:48.664490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:48.664801 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:48.664964 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:48.665138 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 21:04:48.665291 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260927 21:04:48.671051 32558 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:04:48.671674 32558 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:04:48.684794 32558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Generated new TSK 0
I20260927 21:04:48.685070 32558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:04:48.720053 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:48.820170 32572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:48.820410 32572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:48.820436 32572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:48.821739 32572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:48.821798 32572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:48.821826 32572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:04:48.823467 32572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:48.823746 32572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:48.823994 32572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:48.829308 32578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.829335 32577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.829319 32580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:48.829589 32572 server_base.cc:1034] running on GCE node
I20260927 21:04:48.829855 32572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:48.830143 32572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:48.831360 32572 hybrid_clock.cc:648] HybridClock initialized: now 1790543088831327 us; error 58 us; skew 500 ppm
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:48.834163 32572 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:04:48.834681 32572 webserver.cc:460] Webserver started at http://127.16.190.193:33263/ using document root <none> and password file <none>
I20260927 21:04:48.834928 32572 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:48.835007 32572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:48.835158 32572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:48.836272 32572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "5e36dfaf2458497384607f0e82c58f6a"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "9c7f55458ba88fc073a22a1414f5974f"
server_key_iv: "b41fcff6fc2a37290b635387ff4e0902"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.836653 32572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "5e36dfaf2458497384607f0e82c58f6a"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "9c7f55458ba88fc073a22a1414f5974f"
server_key_iv: "b41fcff6fc2a37290b635387ff4e0902"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.838151 32572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:04:48.838948 32587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:48.839133 32572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:04:48.839234 32572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "5e36dfaf2458497384607f0e82c58f6a"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "9c7f55458ba88fc073a22a1414f5974f"
server_key_iv: "b41fcff6fc2a37290b635387ff4e0902"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.839362 32572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:48.850821 32572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:48.853282 32572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:48.853451 32572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:48.853690 32572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:48.854066 32572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:48.854120 32572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:48.854156 32572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:48.854183 32572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:48.860155 32572 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:46487
I20260927 21:04:48.860199 32700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:46487 every 8 connection(s)
I20260927 21:04:48.860548 32572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:04:48.865978 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 32572
I20260927 21:04:48.866096 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:04:48.866315 17147 external_mini_cluster.cc:1428] Setting key b6557f6fa182a5ea5988003e3edfbd65
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:48.871114 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:48.861355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:32779 (local address 127.16.190.254:36853)
0927 21:04:48.861475 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:48.861477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:48.861647 (+ 170us) server_negotiation.cc:407] Connection header received
0927 21:04:48.862205 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:48.862206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:48.862222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:48.862281 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:48.863282 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.863808 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.864963 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.865188 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.865279 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:48.867893 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:48.867894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:48.867894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:48.867903 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:48.869673 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:48.870112 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:48.870113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:48.870114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:48.870177 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:48.870438 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:48.870439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:48.870439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:48.870565 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:48.870637 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:48.870926 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 21:04:48.871013 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 21:04:48.871341 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:48.861265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:32779)
0927 21:04:48.861501 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:48.861551 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:48.862094 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:48.862357 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:48.862362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:48.862973 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:48.863173 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:48.863178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.863908 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.863910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:48.864731 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:48.864733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:48.865291 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:48.865292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:48.865432 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:48.866047 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:48.866089 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:48.867787 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:48.869768 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:48.869776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:48.869779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:48.870019 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:48.870245 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:48.870246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:48.870247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:48.870362 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:48.870640 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:48.870647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:48.870794 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 21:04:48.870904 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:04:48.871089 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
I20260927 21:04:48.872092 32701 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:04:48.872188 32701 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:48.872395 32701 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
I20260927 21:04:48.873005 32510 ts_manager.cc:194] Registered new tserver with Master: 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487)
I20260927 21:04:48.873651 32510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:32779
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:04:48.893966 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:48.977415 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:48.977672 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:48.977715 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:48.978919 32706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:48.978996 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:48.979043 32706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:04:48.980577 32706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:48.980927 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:48.981206 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:48.986943 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.986881 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:48.986963 32711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:48.987190 32706 server_base.cc:1034] running on GCE node
I20260927 21:04:48.987447 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:48.987751 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:48.988909 32706 hybrid_clock.cc:648] HybridClock initialized: now 1790543088988867 us; error 64 us; skew 500 ppm
Sep 27 21:04:48 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:48.991750 32706 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:04:48.992261 32706 webserver.cc:460] Webserver started at http://127.16.190.194:41439/ using document root <none> and password file <none>
I20260927 21:04:48.992553 32706 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:48.992624 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:48.992772 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:48.993901 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cd7cb14d2b9f008149f1ae4c24d0db1c"
server_key_iv: "8822e34c5986afc172b0d60421d34286"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.994287 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cd7cb14d2b9f008149f1ae4c24d0db1c"
server_key_iv: "8822e34c5986afc172b0d60421d34286"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.995792 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:04:48.996801 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:48.997046 32706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:48.997161 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cd7cb14d2b9f008149f1ae4c24d0db1c"
server_key_iv: "8822e34c5986afc172b0d60421d34286"
server_key_version: "encryptionkey@0"
I20260927 21:04:48.997257 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:49.015833 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:49.018471 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:49.018651 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:49.018909 32706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:49.019229 32706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:49.019285 32706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:49.019330 32706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:49.019354 32706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:49.025744 32706 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:39713
I20260927 21:04:49.025810 366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:39713 every 8 connection(s)
I20260927 21:04:49.026166 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:04:49.029040 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 32706
I20260927 21:04:49.029122 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:04:49.029306 17147 external_mini_cluster.cc:1428] Setting key e7569b6701b52aab63db84660efaf136
Sep 27 21:04:49 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543088, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:49.036664 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.026978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:44535 (local address 127.16.190.254:36853)
0927 21:04:49.027056 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.027058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.027278 (+ 220us) server_negotiation.cc:407] Connection header received
0927 21:04:49.027836 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.027838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.027850 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.027907 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:49.028926 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.029632 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.030763 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.030984 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.031060 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.033438 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:49.033439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:49.033440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:49.033449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:49.035103 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.035639 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.035640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.035640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.035696 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.035989 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.035989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.035989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.036101 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:49.036152 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.036518 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.036595 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 21:04:49.037007 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.026944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:44535)
0927 21:04:49.027143 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.027188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.027720 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.027980 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.027984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.028579 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:49.028779 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.028784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.029732 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.029733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.030668 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.030671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.031081 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.031082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.031220 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.031811 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:49.031820 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:49.033352 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:49.035198 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:49.035208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:49.035211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:49.035542 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:49.035762 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:49.035763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:49.035763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:49.035918 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:49.036172 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:49.036180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:49.036367 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.036503 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.036685 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260927 21:04:49.037534 367 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:04:49.037628 367 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:49.037832 367 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
I20260927 21:04:49.038259 32510 ts_manager.cc:194] Registered new tserver with Master: 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713)
I20260927 21:04:49.038709 32510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:44535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:04:49.057421 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:04:49.141489 372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:04:49.141727 372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:04:49.141769 372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:04:49.143057 372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:04:49.143124 372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:04:49.143172 372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:04:49.144762 372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:04:49.145071 372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:04:49.145316 372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:04:49.150696 378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:04:49.150756 377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:49.150797 372 server_base.cc:1034] running on GCE node
W20260927 21:04:49.150775 380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:04:49.151136 372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:04:49.151400 372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:04:49.152611 372 hybrid_clock.cc:648] HybridClock initialized: now 1790543089152595 us; error 44 us; skew 500 ppm
Sep 27 21:04:49 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543089, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:04:49.155546 372 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:04:49.156073 372 webserver.cc:460] Webserver started at http://127.16.190.195:38871/ using document root <none> and password file <none>
I20260927 21:04:49.156356 372 fs_manager.cc:359] Metadata directory not provided
I20260927 21:04:49.156440 372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:04:49.156608 372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:04:49.157871 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "948e9b0336274005b11939b64be527f5"
format_stamp: "Formatted at 2026-09-27 21:04:49 on dist-test-slave-6c7s"
server_key: "4fd65cdca01868d24f4477b1be4f4a78"
server_key_iv: "52edf3e871c0a69c4e945cd494fdd3b8"
server_key_version: "encryptionkey@0"
I20260927 21:04:49.158277 372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "948e9b0336274005b11939b64be527f5"
format_stamp: "Formatted at 2026-09-27 21:04:49 on dist-test-slave-6c7s"
server_key: "4fd65cdca01868d24f4477b1be4f4a78"
server_key_iv: "52edf3e871c0a69c4e945cd494fdd3b8"
server_key_version: "encryptionkey@0"
I20260927 21:04:49.159922 372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:04:49.160867 387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:49.161062 372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:04:49.161167 372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "948e9b0336274005b11939b64be527f5"
format_stamp: "Formatted at 2026-09-27 21:04:49 on dist-test-slave-6c7s"
server_key: "4fd65cdca01868d24f4477b1be4f4a78"
server_key_iv: "52edf3e871c0a69c4e945cd494fdd3b8"
server_key_version: "encryptionkey@0"
I20260927 21:04:49.161262 372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:04:49.173751 372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:04:49.176240 372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:04:49.176419 372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:04:49.176666 372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:04:49.177083 372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:04:49.177140 372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:49.177193 372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:04:49.177227 372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:04:49.183233 372 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:33373
I20260927 21:04:49.183269 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:33373 every 8 connection(s)
I20260927 21:04:49.183629 372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:04:49 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543089, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:04:49.192768 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 372
I20260927 21:04:49.192899 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:04:49.193105 17147 external_mini_cluster.cc:1428] Setting key 65fc76f68a3242f8656e5d9b94656052
I20260927 21:04:49.194064 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.184554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:37935 (local address 127.16.190.254:36853)
0927 21:04:49.184731 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.184738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.184949 (+ 211us) server_negotiation.cc:407] Connection header received
0927 21:04:49.185488 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.185489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.185499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.185557 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:49.186604 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.187130 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.188196 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.188388 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.188447 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.190854 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:49.190856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:49.190856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:49.190863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:49.192296 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.192792 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.192793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.192794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.192905 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.193251 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.193252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.193253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.193423 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:49.193505 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.193856 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.193974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260927 21:04:49.194252 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.184386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:37935)
0927 21:04:49.184773 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.184820 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.185388 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.185633 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.185636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.186226 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:04:49.186472 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.186478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.187215 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.187216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.188113 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.188115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.188469 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.188470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.188580 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.189291 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:04:49.189301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:04:49.190762 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:04:49.192377 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:49.192383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:49.192385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:49.192690 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:49.193007 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:04:49.193009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:04:49.193009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:04:49.193162 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:04:49.193497 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:04:49.193504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:04:49.193711 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.193821 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.193998 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":185,"threads_started":1}
I20260927 21:04:49.194693 501 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:04:49.194811 501 heartbeater.cc:464] Registering TS with master...
I20260927 21:04:49.194999 501 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
I20260927 21:04:49.195398 32510 ts_manager.cc:194] Registered new tserver with Master: 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
I20260927 21:04:49.195780 32510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:37935
I20260927 21:04:49.205159 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:04:49.212740 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.206645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.16.190.254:36853)
0927 21:04:49.206722 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.206724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.206834 (+ 110us) server_negotiation.cc:407] Connection header received
0927 21:04:49.206863 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.206863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.206883 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.206952 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:04:49.207487 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.208004 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.208762 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.209062 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.209133 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.210479 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:04:49.210480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:04:49.210481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:04:49.210487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:04:49.211755 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.212072 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.212073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.212073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.212126 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:04:49.212278 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:04:49.212279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:04:49.212279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:04:49.212427 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:04:49.212507 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.212609 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.212670 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 21:04:49.214502 32510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:04:49.216177 32510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:04:49.222815 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.218746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.16.190.194:39713)
0927 21:04:49.219091 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.219097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.219107 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:04:49.219152 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.219154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.219170 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.219247 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:49.219532 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220252 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.221631 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222373 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222450 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.222547 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.222673 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.222714 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":82,"threads_started":1}
I20260927 21:04:49.222800 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.218650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39713 (local address 127.0.0.1:33518)
0927 21:04:49.218953 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.218966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.219029 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.219356 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.219359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.219362 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:49.219434 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.219436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220366 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.220367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.221544 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.221546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222474 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.222562 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.222577 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.222652 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.222709 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":82,"threads_started":1}
I20260927 21:04:49.222995 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.219078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.16.190.195:33373)
0927 21:04:49.219289 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.219293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.219302 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:49.219338 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.219340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.219357 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.219426 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:49.219619 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220746 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.221965 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222464 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222534 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.222602 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.222841 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.222888 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260927 21:04:49.223073 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.218847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:33373 (local address 127.0.0.1:50672)
0927 21:04:49.219053 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.219062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.219102 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.219501 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.219502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.219504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:49.219544 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.219545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220898 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.220899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.221880 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.221882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222634 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.222686 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.222689 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.222733 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.222763 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":27,"thread_start_us":69,"threads_started":1}
I20260927 21:04:49.223160 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.218646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:46487 (local address 127.0.0.1:44306)
0927 21:04:49.218819 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.218863 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.218934 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.219437 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.219438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.219440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:49.219492 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.219494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220891 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.220892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.221951 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.221954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222848 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.222912 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.222917 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.222953 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.223022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":53,"threads_started":1}
I20260927 21:04:49.223798 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.218755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.16.190.193:46487)
0927 21:04:49.219250 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.219254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.219265 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:04:49.219310 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.219313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.219329 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.219584 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:49.219665 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.220743 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.222052 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.222750 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.223570 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.223638 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.223665 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.223710 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":89,"threads_started":1}
I20260927 21:04:49.223726 301 tablet_service.cc:1467] Processing CreateTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 (DEFAULT_TABLE table=test-table [id=3cb925d4957149e0a52b2a70f769dfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:49.224103 301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fdffbb71cd427898c8eb06c1032c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:49.224225 435 tablet_service.cc:1467] Processing CreateTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 (DEFAULT_TABLE table=test-table [id=3cb925d4957149e0a52b2a70f769dfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:49.224529 435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fdffbb71cd427898c8eb06c1032c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:49.224551 32635 tablet_service.cc:1467] Processing CreateTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 (DEFAULT_TABLE table=test-table [id=3cb925d4957149e0a52b2a70f769dfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:04:49.224947 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fdffbb71cd427898c8eb06c1032c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:04:49.227669 518 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Bootstrap starting.
I20260927 21:04:49.227921 520 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Bootstrap starting.
I20260927 21:04:49.228106 519 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap starting.
I20260927 21:04:49.228415 518 tablet_bootstrap.cc:654] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:49.228570 520 tablet_bootstrap.cc:654] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:49.228747 519 tablet_bootstrap.cc:654] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:04:49.229106 518 log.cc:824] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:49.229156 520 log.cc:824] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:49.229333 519 log.cc:824] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:04:49.229961 520 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: No bootstrap required, opened a new log
I20260927 21:04:49.230038 520 ts_tablet_manager.cc:1397] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:04:49.230178 518 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: No bootstrap required, opened a new log
I20260927 21:04:49.230238 518 ts_tablet_manager.cc:1397] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 21:04:49.231442 520 raft_consensus.cc:348] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.231590 520 raft_consensus.cc:374] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:49.231613 520 raft_consensus.cc:729] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948e9b0336274005b11939b64be527f5, State: Initialized, Role: FOLLOWER
I20260927 21:04:49.231729 520 consensus_queue.cc:260] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.231649 518 raft_consensus.cc:348] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.231850 518 raft_consensus.cc:374] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:49.231894 518 raft_consensus.cc:729] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9b638400ca444aa7a5d8424fb9b50e, State: Initialized, Role: FOLLOWER
I20260927 21:04:49.231998 518 consensus_queue.cc:260] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.232018 501 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
I20260927 21:04:49.232023 520 ts_tablet_manager.cc:1428] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:04:49.232271 518 ts_tablet_manager.cc:1428] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:04:49.232307 367 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
I20260927 21:04:49.233155 519 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: No bootstrap required, opened a new log
I20260927 21:04:49.233233 519 ts_tablet_manager.cc:1397] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 21:04:49.234824 519 raft_consensus.cc:348] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.234967 519 raft_consensus.cc:374] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:04:49.234990 519 raft_consensus.cc:729] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36dfaf2458497384607f0e82c58f6a, State: Initialized, Role: FOLLOWER
I20260927 21:04:49.235133 519 consensus_queue.cc:260] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.235414 519 ts_tablet_manager.cc:1428] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:04:49.235443 32701 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
W20260927 21:04:49.276787 368 tablet.cc:2369] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:04:49.361378 32702 tablet.cc:2369] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:04:49.381060 526 raft_consensus.cc:482] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:04:49.381202 526 raft_consensus.cc:504] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.381547 526 leader_election.cc:288] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
I20260927 21:04:49.386330 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.381982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:60039 (local address 127.16.190.194:39713)
0927 21:04:49.382035 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.382038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.382050 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:04:49.382123 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.382124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.382136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.382218 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:49.382607 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.383430 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385203 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.385802 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385891 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.385901 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.386203 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.386253 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:04:49.386392 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.382014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:35161 (local address 127.16.190.195:33373)
0927 21:04:49.382058 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.382060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.382069 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:04:49.382123 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.382124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.382134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.382218 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:04:49.382606 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.383366 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385203 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.385802 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385891 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.385901 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.386203 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.386255 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:04:49.386343 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.381811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:33373 (local address 127.16.190.193:35161)
0927 21:04:49.381972 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.381987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.382056 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.382270 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.382272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.382275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:49.382420 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.382422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.383535 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.383536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.385068 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.385071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.385959 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.386061 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.386069 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.386145 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.386208 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":152,"thread_start_us":85,"threads_started":1}
I20260927 21:04:49.386343 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.381813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39713 (local address 127.16.190.193:60039)
0927 21:04:49.381972 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.381987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:04:49.382056 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:04:49.382270 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:04:49.382272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:04:49.382276 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:04:49.382417 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.382419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.383535 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.383536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.385066 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:04:49.385071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.385961 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.385962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:04:49.386061 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.386071 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 21:04:49.386144 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:04:49.386208 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":172,"thread_start_us":60,"threads_started":1}
I20260927 21:04:49.386822 321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fdffbb71cd427898c8eb06c1032c27" candidate_uuid: "5e36dfaf2458497384607f0e82c58f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" is_pre_election: true
I20260927 21:04:49.386837 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fdffbb71cd427898c8eb06c1032c27" candidate_uuid: "5e36dfaf2458497384607f0e82c58f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948e9b0336274005b11939b64be527f5" is_pre_election: true
I20260927 21:04:49.387017 455 raft_consensus.cc:2454] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e36dfaf2458497384607f0e82c58f6a in term 0.
I20260927 21:04:49.387017 321 raft_consensus.cc:2454] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e36dfaf2458497384607f0e82c58f6a in term 0.
I20260927 21:04:49.387300 32589 leader_election.cc:302] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a, 9b9b638400ca444aa7a5d8424fb9b50e; no voters:
I20260927 21:04:49.387496 526 raft_consensus.cc:2781] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:04:49.387575 526 raft_consensus.cc:482] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:04:49.387593 526 raft_consensus.cc:3037] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:49.388418 526 raft_consensus.cc:504] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.388577 526 leader_election.cc:288] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 1 election: Requested vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
I20260927 21:04:49.388777 455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fdffbb71cd427898c8eb06c1032c27" candidate_uuid: "5e36dfaf2458497384607f0e82c58f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948e9b0336274005b11939b64be527f5"
I20260927 21:04:49.388937 455 raft_consensus.cc:3037] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:49.389712 455 raft_consensus.cc:2454] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e36dfaf2458497384607f0e82c58f6a in term 1.
I20260927 21:04:49.389880 321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fdffbb71cd427898c8eb06c1032c27" candidate_uuid: "5e36dfaf2458497384607f0e82c58f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
I20260927 21:04:49.389948 32590 leader_election.cc:302] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a, 948e9b0336274005b11939b64be527f5; no voters:
I20260927 21:04:49.389969 321 raft_consensus.cc:3037] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:04:49.390043 526 raft_consensus.cc:2781] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:04:49.390223 526 raft_consensus.cc:686] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 LEADER]: Becoming Leader. State: Replica: 5e36dfaf2458497384607f0e82c58f6a, State: Running, Role: LEADER
I20260927 21:04:49.390339 526 consensus_queue.cc:237] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:04:49.390887 321 raft_consensus.cc:2454] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e36dfaf2458497384607f0e82c58f6a in term 1.
I20260927 21:04:49.391533 32510 catalog_manager.cc:5495] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193). New cstate: current_term: 1 leader_uuid: "5e36dfaf2458497384607f0e82c58f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } health_report { overall_health: UNKNOWN } } }
I20260927 21:04:49.404929 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.402417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.16.190.193:46487)
0927 21:04:49.402483 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:04:49.402485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:04:49.402498 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:04:49.402535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:04:49.402536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:04:49.402554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:04:49.402616 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:04:49.402776 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.403270 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.404031 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:04:49.404279 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:04:49.404393 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:04:49.404528 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:04:49.404636 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:04:49.404709 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:04:49.404740 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 21:04:49.404758 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:04:49.406809 321 raft_consensus.cc:1261] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Refusing update from remote peer 5e36dfaf2458497384607f0e82c58f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:49.406841 455 raft_consensus.cc:1261] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Refusing update from remote peer 5e36dfaf2458497384607f0e82c58f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:04:49.407245 529 consensus_queue.cc:1035] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:49.407389 526 consensus_queue.cc:1035] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:04:49.409994 534 mvcc.cc:205] Tried to move back new op lower bound from 7334064494207545344 to 7334064494143557632. Current Snapshot: MvccSnapshot[applied={T|T < 7334064494207545344}]
I20260927 21:04:49.412484 32510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:49.412593 32510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:04:49.413540 32510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Sending DeleteTablet for 3 replicas of tablet a1fdffbb71cd427898c8eb06c1032c27
I20260927 21:04:49.413914 301 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:49 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:33518
I20260927 21:04:49.413941 32635 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:49 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:44306
I20260927 21:04:49.413964 435 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:49 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:50672
I20260927 21:04:49.414170 543 tablet_replica.cc:330] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: stopping tablet replica
I20260927 21:04:49.414206 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 32572
I20260927 21:04:49.414300 542 tablet_replica.cc:330] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: stopping tablet replica
I20260927 21:04:49.414379 542 raft_consensus.cc:2229] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:49.414475 542 raft_consensus.cc:2258] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:49.416054 544 tablet_replica.cc:330] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: stopping tablet replica
I20260927 21:04:49.416124 542 ts_tablet_manager.cc:1905] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:49.416188 544 raft_consensus.cc:2229] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:04:49.416303 544 raft_consensus.cc:2258] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:04:49.417822 544 ts_tablet_manager.cc:1905] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:04:49.418092 542 ts_tablet_manager.cc:1918] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:49.418182 542 log.cc:1196] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/a1fdffbb71cd427898c8eb06c1032c27
I20260927 21:04:49.418550 542 ts_tablet_manager.cc:1939] T a1fdffbb71cd427898c8eb06c1032c27 P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting consensus metadata
I20260927 21:04:49.419055 32495 catalog_manager.cc:4841] TS 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): tablet a1fdffbb71cd427898c8eb06c1032c27 (table test-table [id=3cb925d4957149e0a52b2a70f769dfad]) successfully deleted
I20260927 21:04:49.419829 544 ts_tablet_manager.cc:1918] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:04:49.419917 544 log.cc:1196] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/a1fdffbb71cd427898c8eb06c1032c27
I20260927 21:04:49.420205 544 ts_tablet_manager.cc:1939] T a1fdffbb71cd427898c8eb06c1032c27 P 948e9b0336274005b11939b64be527f5: Deleting consensus metadata
I20260927 21:04:49.420686 32496 catalog_manager.cc:4841] TS 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): tablet a1fdffbb71cd427898c8eb06c1032c27 (table test-table [id=3cb925d4957149e0a52b2a70f769dfad]) successfully deleted
W20260927 21:04:49.420910 32496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:46487 (error 108)
I20260927 21:04:49.421103 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 32706
I20260927 21:04:49.421764 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:04:49.421322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:46487 (local address 127.0.0.1:44208)
0927 21:04:49.421684 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 21:04:49.421723 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
W20260927 21:04:49.422262 32496 catalog_manager.cc:4584] TS 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1fdffbb71cd427898c8eb06c1032c27: Network error: Client connection negotiation failed: client connection to 127.16.190.193:46487: connect: Connection refused (error 111)
I20260927 21:04:49.426048 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 372
I20260927 21:04:49.430826 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 32478
I20260927 21:05:09.436437 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--webserver_interface=127.16.190.254
--webserver_port=45365
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:09.520083 546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:09.520345 546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:09.520385 546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:09.521591 546 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:05:09.521667 546 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:05:09.521689 546 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:05:09.521709 546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:09.521742 546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:09.521767 546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:09.521793 546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:09.521821 546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:09.523341 546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:09.523628 546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:09.523857 546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:09.533476 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.533414 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.533418 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:09.533587 546 server_base.cc:1034] running on GCE node
I20260927 21:05:09.534034 546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:09.534333 546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:09.535480 546 hybrid_clock.cc:648] HybridClock initialized: now 1790543109535456 us; error 36 us; skew 500 ppm
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:09.538125 546 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:05:09.538677 546 webserver.cc:460] Webserver started at http://127.16.190.254:45365/ using document root <none> and password file <none>
I20260927 21:05:09.538971 546 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:09.539063 546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:09.540525 546 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:09.541404 562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.541620 546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:05:09.541725 546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a6229a26c9914480bc03c6a5d0a6dec9"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cbe1ad79ab065ab6971dae9c2d1498ac"
server_key_iv: "ddaa78d6d6cdcecb7aeb1276c2901baf"
server_key_version: "encryptionkey@0"
I20260927 21:05:09.542183 546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:09.554209 546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:09.556368 546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:09.556669 546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:09.560942 546 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36853
I20260927 21:05:09.560979 614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36853 every 8 connection(s)
I20260927 21:05:09.561393 546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:09.561656 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 546
I20260927 21:05:09.561939 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:46487
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=33263
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:05:09.563076 615 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:05:09.563788 615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap starting.
I20260927 21:05:09.575824 615 log.cc:824] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:09.578441 615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:05:09.578802 615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap complete.
I20260927 21:05:09.582261 615 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:09.582593 615 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Initialized, Role: FOLLOWER
I20260927 21:05:09.582754 615 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:09.582818 615 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:09.582861 615 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:09.582906 615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 21:05:09.584259 615 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:09.584369 615 leader_election.cc:302] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6229a26c9914480bc03c6a5d0a6dec9; no voters:
I20260927 21:05:09.584532 615 leader_election.cc:288] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 21:05:09.584617 619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 21:05:09.584779 615 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:09.584870 619 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 LEADER]: Becoming Leader. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Running, Role: LEADER
I20260927 21:05:09.585074 619 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:09.585383 620 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:05:09.585383 619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6229a26c9914480bc03c6a5d0a6dec9. Latest consensus state: current_term: 2 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:05:09.585623 619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:09.585834 620 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:09.585824 628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:09.586927 628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:09.587051 628 catalog_manager.cc:1259] Loaded cluster ID: b77672ebc59743ae92aeb3bfeb774ac3
I20260927 21:05:09.587122 628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:09.587940 628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:09.588131 628 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Loaded TSK: 0
I20260927 21:05:09.597543 628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Generated new TSK 1
I20260927 21:05:09.597682 628 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:05:09.663201 617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:09.663486 617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:09.663527 617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:09.664913 617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:09.664989 617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:09.665036 617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:09.666610 617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:46487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=33263
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:09.666900 617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:09.667150 617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:09.673372 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.673493 645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.673352 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:09.673622 617 server_base.cc:1034] running on GCE node
I20260927 21:05:09.673820 617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:09.674098 617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:09.675272 617 hybrid_clock.cc:648] HybridClock initialized: now 1790543109675260 us; error 29 us; skew 500 ppm
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:09.678005 617 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:05:09.678539 617 webserver.cc:460] Webserver started at http://127.16.190.193:33263/ using document root <none> and password file <none>
I20260927 21:05:09.678784 617 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:09.678848 617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:09.680291 617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:09.681074 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.681269 617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:09.681382 617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "5e36dfaf2458497384607f0e82c58f6a"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "9c7f55458ba88fc073a22a1414f5974f"
server_key_iv: "b41fcff6fc2a37290b635387ff4e0902"
server_key_version: "encryptionkey@0"
I20260927 21:05:09.681764 617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:09.692993 617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:09.695518 617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:09.695698 617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:09.695972 617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:09.696463 659 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:05:09.698119 617 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:05:09.698179 617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:05:09.698205 617 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:05:09.698935 617 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:05:09.698979 617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:05:09.699030 659 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap starting.
I20260927 21:05:09.707449 617 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:46487
I20260927 21:05:09.708201 617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:09.708676 766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:46487 every 8 connection(s)
I20260927 21:05:09.711215 659 log.cc:824] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:09.713796 659 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:05:09.714138 659 tablet_bootstrap.cc:492] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap complete.
I20260927 21:05:09.715133 659 ts_tablet_manager.cc:1397] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260927 21:05:09.716476 659 raft_consensus.cc:348] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:05:09.716830 659 raft_consensus.cc:729] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36dfaf2458497384607f0e82c58f6a, State: Initialized, Role: FOLLOWER
I20260927 21:05:09.717003 659 consensus_queue.cc:260] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } }
I20260927 21:05:09.717350 659 ts_tablet_manager.cc:1428] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260927 21:05:09.718277 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 617
I20260927 21:05:09.718557 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:39713
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=41439
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:09.725589 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.709098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:49247)
0927 21:05:09.709506 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:09.709554 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:09.710186 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:09.713486 (+ 3300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:09.713491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:09.713862 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:09.714170 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.714179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.715889 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.715890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.717163 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.717166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.717935 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.717936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.718073 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.718851 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:09.718860 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:09.720551 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:09.723070 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.723084 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.723088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.723472 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.723919 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.723920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.723921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.724096 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.724734 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:09.724744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:09.724970 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 21:05:09.725086 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:05:09.725343 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":88,"threads_started":1}
I20260927 21:05:09.726722 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.709524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:49247 (local address 127.16.190.254:36853)
0927 21:05:09.712968 (+ 3444us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:09.712978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:09.713029 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:05:09.713178 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:09.713185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:09.713246 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:09.713421 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:09.714320 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.715766 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.717290 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.717791 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.717967 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.720702 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:09.720707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:09.720708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:09.720722 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:09.722917 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.723587 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.723591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.723591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.723728 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.724190 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.724191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.724191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.724612 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:09.726260 (+ 1648us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:09.726353 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 21:05:09.726530 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3385,"thread_start_us":116,"threads_started":1}
I20260927 21:05:09.727717 767 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:05:09.727813 767 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:09.727993 767 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
I20260927 21:05:09.728920 579 ts_manager.cc:194] Registered new tserver with Master: 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487)
I20260927 21:05:09.730125 579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:49247
I20260927 21:05:09.731139 767 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.193@KRBTEST.COM
I20260927 21:05:09.739413 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.729579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:46487 (local address 127.0.0.1:56168)
0927 21:05:09.730112 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:09.730150 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:09.730249 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:09.730503 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:09.730506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:09.730857 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:09.730932 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.730934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.732258 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.732259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.733284 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.733287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.733938 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.733939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.734020 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.734593 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:09.734599 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:09.736223 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:09.738129 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.738136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.738139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.738353 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.738589 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.738590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.738590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.738660 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.739017 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:09.739023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:09.739067 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:05:09.739162 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 21:05:09.739297 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":97,"threads_started":1}
I20260927 21:05:09.739405 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.729867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.16.190.193:46487)
0927 21:05:09.730037 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:09.730041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:09.730210 (+ 169us) server_negotiation.cc:407] Connection header received
0927 21:05:09.730351 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:09.730353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:09.730370 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:09.730412 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:09.731027 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.732158 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.733430 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.733848 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.733929 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.736326 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:09.736336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:09.736336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:09.736344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:09.738017 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.738448 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.738451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.738452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.738516 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.738725 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.738726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.738726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.738884 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:09.738981 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:09.739199 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 21:05:09.739297 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260927 21:05:09.740047 701 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fdffbb71cd427898c8eb06c1032c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:04:49 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56168
I20260927 21:05:09.740556 776 tablet_replica.cc:330] stopping tablet replica
I20260927 21:05:09.740692 776 raft_consensus.cc:2229] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:09.740790 776 raft_consensus.cc:2258] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:09.741127 776 ts_tablet_manager.cc:1905] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:09.743520 776 ts_tablet_manager.cc:1918] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:09.743613 776 log.cc:1196] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal/wals/a1fdffbb71cd427898c8eb06c1032c27
I20260927 21:05:09.743832 776 ts_tablet_manager.cc:1939] T a1fdffbb71cd427898c8eb06c1032c27 P 5e36dfaf2458497384607f0e82c58f6a: Deleting consensus metadata
I20260927 21:05:09.744366 565 catalog_manager.cc:4841] TS 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487): tablet a1fdffbb71cd427898c8eb06c1032c27 (table test-table [id=3cb925d4957149e0a52b2a70f769dfad]) successfully deleted
W20260927 21:05:09.808348 773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:09.808571 773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:09.808596 773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:09.809746 773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:09.809793 773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:09.809820 773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:09.811214 773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:39713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=41439
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:09.811410 773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:09.811594 773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:09.816696 784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.816730 782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.816748 781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:09.816908 773 server_base.cc:1034] running on GCE node
I20260927 21:05:09.817134 773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:09.817394 773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:09.818586 773 hybrid_clock.cc:648] HybridClock initialized: now 1790543109818580 us; error 39 us; skew 500 ppm
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:09.821467 773 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:05:09.821949 773 webserver.cc:460] Webserver started at http://127.16.190.194:41439/ using document root <none> and password file <none>
I20260927 21:05:09.822206 773 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:09.822291 773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:09.823706 773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:09.824397 791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.824571 773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:09.824682 773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cd7cb14d2b9f008149f1ae4c24d0db1c"
server_key_iv: "8822e34c5986afc172b0d60421d34286"
server_key_version: "encryptionkey@0"
I20260927 21:05:09.825165 773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:09.852020 773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:09.854666 773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:09.854833 773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:09.855109 773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:09.855427 773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:09.855460 773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.855484 773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:09.855495 773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.862046 773 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:39713
I20260927 21:05:09.862130 905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:39713 every 8 connection(s)
I20260927 21:05:09.862509 773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:09.864848 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 773
I20260927 21:05:09.865140 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:33373
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=38871
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:09.873538 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.863286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:33699 (local address 127.16.190.254:36853)
0927 21:05:09.863329 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:09.863331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:09.863571 (+ 240us) server_negotiation.cc:407] Connection header received
0927 21:05:09.864063 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:09.864064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:09.864078 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:09.864130 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:09.865308 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.865817 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.867313 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.867523 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.867587 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.870127 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:09.870129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:09.870130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:09.870138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:09.871866 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.872377 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.872379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.872379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.872452 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.872742 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.872743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.872743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.872939 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:09.873025 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:09.873327 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 21:05:09.873427 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:05:09.873730 908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.863292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:33699)
0927 21:05:09.863442 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:09.863478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:09.863963 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:09.864196 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:09.864199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:09.864899 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:09.865155 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.865160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.865921 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.865922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.867210 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.867214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.867614 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.867615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.867756 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.868353 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:09.868364 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:09.870007 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:09.872010 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.872019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.872021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.872280 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.872534 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.872535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.872536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.872663 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.873014 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:09.873022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:09.873177 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 21:05:09.873290 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 21:05:09.873465 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260927 21:05:09.874253 906 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:05:09.874352 906 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:09.874572 906 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
I20260927 21:05:09.875008 579 ts_manager.cc:194] Registered new tserver with Master: 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713)
I20260927 21:05:09.875586 579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:33699
W20260927 21:05:09.949611 909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:09.949828 909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:09.949851 909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:09.950958 909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:09.950995 909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:09.951021 909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:09.952399 909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:33373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=38871
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:09.952601 909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:09.952786 909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:09.957939 917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.957939 914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:09.957928 915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:09.958241 909 server_base.cc:1034] running on GCE node
I20260927 21:05:09.958527 909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:09.958788 909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:09.959960 909 hybrid_clock.cc:648] HybridClock initialized: now 1790543109959939 us; error 40 us; skew 500 ppm
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:09.962623 909 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:05:09.963085 909 webserver.cc:460] Webserver started at http://127.16.190.195:38871/ using document root <none> and password file <none>
I20260927 21:05:09.963331 909 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:09.963414 909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:09.964847 909 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:05:09.965479 924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.965678 909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:09.965747 909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "948e9b0336274005b11939b64be527f5"
format_stamp: "Formatted at 2026-09-27 21:04:49 on dist-test-slave-6c7s"
server_key: "4fd65cdca01868d24f4477b1be4f4a78"
server_key_iv: "52edf3e871c0a69c4e945cd494fdd3b8"
server_key_version: "encryptionkey@0"
I20260927 21:05:09.966120 909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:09.979415 909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:09.981627 909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:09.981797 909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:09.982034 909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:09.982352 909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:09.982405 909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.982450 909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:09.982479 909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:09.988860 909 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:33373
I20260927 21:05:09.988940 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:33373 every 8 connection(s)
I20260927 21:05:09.989319 909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:09.990806 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 909
Sep 27 21:05:09 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:09.998193 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.991349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.16.190.254:36853)
0927 21:05:09.991494 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:09.991496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:09.991504 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:09.991525 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:09.991526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:09.991538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:09.991625 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:09.992252 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.992768 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.993694 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.993903 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.993954 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.995314 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:09.995316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:09.995317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:09.995324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:09.997026 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.997461 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.997462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.997462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.997515 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.997725 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.997726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.997726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.997829 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:09.997965 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:09.998056 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:05:09.998120 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260927 21:05:09.999591 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.990093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:54681 (local address 127.16.190.254:36853)
0927 21:05:09.990167 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:09.990169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:09.990362 (+ 193us) server_negotiation.cc:407] Connection header received
0927 21:05:09.990901 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:09.990902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:09.990916 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:09.990966 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:09.992064 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.992641 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.993748 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.993990 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.994068 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.996358 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:09.996359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:09.996359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:09.996367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:09.998041 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.998534 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.998535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.998536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.998597 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:09.998869 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:09.998870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:09.998870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:09.999025 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:09.999102 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:09.999389 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 21:05:09.999505 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:05:09.999788 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:09.990088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:54681)
0927 21:05:09.990231 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:09.990274 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:09.990775 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:09.991077 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:09.991082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:09.991756 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:09.991959 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.991965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.992728 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.992729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.993619 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:09.993621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:09.994077 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:09.994078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:09.994182 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:09.994777 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:09.994786 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:09.996249 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:09.998123 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.998130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.998133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.998439 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.998668 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:09.998669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:09.998670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:09.998803 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:09.999095 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:09.999102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:09.999250 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 21:05:09.999369 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:05:09.999540 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260927 21:05:10.000260 1038 heartbeater.cc:347] Connected to a master server at 127.16.190.254:36853
I20260927 21:05:10.000348 1038 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:10.000502 1038 heartbeater.cc:511] Master 127.16.190.254:36853 requested a full tablet report, sending...
I20260927 21:05:10.000871 579 ts_manager.cc:194] Registered new tserver with Master: 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
I20260927 21:05:10.001261 579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:54681
I20260927 21:05:10.010207 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:10.013352 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.010647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.16.190.254:36853)
0927 21:05:10.010731 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.010734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.010845 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:05:10.010964 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.010967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.011012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.011095 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:10.011275 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.011768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.012561 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.012754 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.012854 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.013034 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:10.013124 (+ 90us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 21:05:10.013298 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44860: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 21:05:10.013437 770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44860: token expired
W20260927 21:05:10.013795 504 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.190.254:36853)
I20260927 21:05:10.021234 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.014117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.16.190.254:36853)
0927 21:05:10.014177 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.014179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.014300 (+ 121us) server_negotiation.cc:407] Connection header received
0927 21:05:10.014331 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.014332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.014343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.014422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:10.014898 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.015678 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.016476 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.016714 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.016767 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.018416 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:10.018418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:10.018418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:10.018424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:10.019939 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:10.020360 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:10.020360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:10.020360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:10.020432 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:10.020640 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:10.020642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:10.020642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:10.020769 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:10.020913 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.021021 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.021103 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 21:05:10.022838 579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:10.025802 579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:10.029139 701 tablet_service.cc:1467] Processing CreateTablet for tablet dbab9dd64c64410dafe8e747fff82a6d (DEFAULT_TABLE table=test-table [id=7e57b8eddca14d84aa3d70c672617f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:10.029371 701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbab9dd64c64410dafe8e747fff82a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:10.031077 659 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap starting.
I20260927 21:05:10.031381 659 tablet_bootstrap.cc:654] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:10.032433 659 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: No bootstrap required, opened a new log
I20260927 21:05:10.032490 659 ts_tablet_manager.cc:1397] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:10.032641 659 raft_consensus.cc:348] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.032732 659 raft_consensus.cc:374] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:10.032778 659 raft_consensus.cc:729] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36dfaf2458497384607f0e82c58f6a, State: Initialized, Role: FOLLOWER
I20260927 21:05:10.032862 659 consensus_queue.cc:260] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.032968 659 ts_tablet_manager.cc:1428] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:10.033074 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.028375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39713 (local address 127.0.0.1:55494)
0927 21:05:10.028612 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:10.028650 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:10.028703 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:10.029723 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:10.029724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:10.029726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:10.029772 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.029774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.031000 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.031002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.031807 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.031809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.032770 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.032771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.032862 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.032887 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 21:05:10.032948 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 21:05:10.032995 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 21:05:10.033113 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.028516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.16.190.194:39713)
0927 21:05:10.029451 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.029456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.029467 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:10.029509 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.029512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.029527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.029609 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:10.029847 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.030900 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.031910 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.032662 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.032737 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.032801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.032979 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.033030 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":102,"threads_started":1}
I20260927 21:05:10.033496 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.028440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:33373 (local address 127.0.0.1:56504)
0927 21:05:10.029189 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:10.029199 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:10.029259 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:10.029708 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:10.029709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:10.029711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:10.029764 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.029765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.031067 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.031068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.032217 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.032220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.033223 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.033224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.033301 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.033305 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:10.033368 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:05:10.033415 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":105,"threads_started":1}
I20260927 21:05:10.033516 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.029020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.16.190.195:33373)
0927 21:05:10.029515 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.029520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.029532 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:10.029572 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.029575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.029590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.029835 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:10.029922 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.030957 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.032391 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.033151 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.033227 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.033288 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.033379 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.033435 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":84,"threads_started":1}
I20260927 21:05:10.033885 840 tablet_service.cc:1467] Processing CreateTablet for tablet dbab9dd64c64410dafe8e747fff82a6d (DEFAULT_TABLE table=test-table [id=7e57b8eddca14d84aa3d70c672617f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:10.034296 840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbab9dd64c64410dafe8e747fff82a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:10.034566 972 tablet_service.cc:1467] Processing CreateTablet for tablet dbab9dd64c64410dafe8e747fff82a6d (DEFAULT_TABLE table=test-table [id=7e57b8eddca14d84aa3d70c672617f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:10.034884 972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbab9dd64c64410dafe8e747fff82a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:10.038177 1049 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Bootstrap starting.
I20260927 21:05:10.038271 1050 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Bootstrap starting.
I20260927 21:05:10.038967 1050 tablet_bootstrap.cc:654] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:10.038966 1049 tablet_bootstrap.cc:654] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:10.039475 1050 log.cc:824] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:10.039556 1049 log.cc:824] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:10.040237 1050 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: No bootstrap required, opened a new log
I20260927 21:05:10.040269 1049 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: No bootstrap required, opened a new log
I20260927 21:05:10.040318 1049 ts_tablet_manager.cc:1397] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:10.040361 1050 ts_tablet_manager.cc:1397] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:10.041990 1049 raft_consensus.cc:348] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.042012 1050 raft_consensus.cc:348] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.042145 1049 raft_consensus.cc:374] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:10.042171 1049 raft_consensus.cc:729] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9b638400ca444aa7a5d8424fb9b50e, State: Initialized, Role: FOLLOWER
I20260927 21:05:10.042171 1050 raft_consensus.cc:374] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:10.042214 1050 raft_consensus.cc:729] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948e9b0336274005b11939b64be527f5, State: Initialized, Role: FOLLOWER
I20260927 21:05:10.042308 1049 consensus_queue.cc:260] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.042336 1050 consensus_queue.cc:260] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.042639 1050 ts_tablet_manager.cc:1428] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:05:10.042632 1049 ts_tablet_manager.cc:1428] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:10.042646 906 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
I20260927 21:05:10.042748 1038 heartbeater.cc:503] Master 127.16.190.254:36853 was elected leader, sending a full tablet report...
I20260927 21:05:10.078014 1054 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:10.078142 1054 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.078506 1054 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487)
I20260927 21:05:10.081571 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.078670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39713 (local address 127.16.190.195:47713)
0927 21:05:10.078927 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:10.078937 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:10.079004 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:10.079133 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:10.079134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:10.079135 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:10.079227 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.079228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.079827 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.079828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.080679 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.080681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.081279 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.081280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.081357 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.081362 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:10.081415 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:05:10.081456 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":101,"thread_start_us":103,"threads_started":1}
I20260927 21:05:10.081578 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.078720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:47713 (local address 127.16.190.194:39713)
0927 21:05:10.078778 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.078780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.079013 (+ 233us) server_negotiation.cc:407] Connection header received
0927 21:05:10.079049 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.079050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.079062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.079121 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:10.079293 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.079740 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.080770 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.081198 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.081259 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.081265 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.081444 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.081484 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:05:10.081720 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.078763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:46591 (local address 127.16.190.193:46487)
0927 21:05:10.078807 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.078809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.078817 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:10.078875 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.078876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.078904 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.078997 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:10.079210 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.079763 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.080853 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.081303 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.081379 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.081467 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.081581 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.081634 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:05:10.081670 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.078669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:46487 (local address 127.16.190.195:46591)
0927 21:05:10.078726 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:10.078739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:10.078791 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:10.079036 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:10.079038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:10.079039 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:10.079118 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.079120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.079875 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.079876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.080682 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:10.080685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.081390 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.081421 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:10.081499 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.081502 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:10.081562 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:05:10.081600 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":49}
I20260927 21:05:10.082159 860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbab9dd64c64410dafe8e747fff82a6d" candidate_uuid: "948e9b0336274005b11939b64be527f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" is_pre_election: true
I20260927 21:05:10.082226 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbab9dd64c64410dafe8e747fff82a6d" candidate_uuid: "948e9b0336274005b11939b64be527f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36dfaf2458497384607f0e82c58f6a" is_pre_election: true
I20260927 21:05:10.082340 860 raft_consensus.cc:2454] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948e9b0336274005b11939b64be527f5 in term 0.
I20260927 21:05:10.082346 721 raft_consensus.cc:2454] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 948e9b0336274005b11939b64be527f5 in term 0.
I20260927 21:05:10.082577 926 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e; no voters:
I20260927 21:05:10.082713 1054 raft_consensus.cc:2781] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:10.082770 1054 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:10.082800 1054 raft_consensus.cc:3037] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:10.083513 1054 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.083662 1054 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487)
I20260927 21:05:10.083818 860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbab9dd64c64410dafe8e747fff82a6d" candidate_uuid: "948e9b0336274005b11939b64be527f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
I20260927 21:05:10.083818 721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbab9dd64c64410dafe8e747fff82a6d" candidate_uuid: "948e9b0336274005b11939b64be527f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e36dfaf2458497384607f0e82c58f6a"
I20260927 21:05:10.083894 721 raft_consensus.cc:3037] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:10.083894 860 raft_consensus.cc:3037] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:10.084800 721 raft_consensus.cc:2454] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948e9b0336274005b11939b64be527f5 in term 1.
I20260927 21:05:10.084830 860 raft_consensus.cc:2454] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948e9b0336274005b11939b64be527f5 in term 1.
I20260927 21:05:10.084985 927 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a, 948e9b0336274005b11939b64be527f5; no voters:
I20260927 21:05:10.085080 1054 raft_consensus.cc:2781] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:10.085244 1054 raft_consensus.cc:686] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 1 LEADER]: Becoming Leader. State: Replica: 948e9b0336274005b11939b64be527f5, State: Running, Role: LEADER
I20260927 21:05:10.085354 1054 consensus_queue.cc:237] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:10.086416 578 catalog_manager.cc:5495] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 948e9b0336274005b11939b64be527f5 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "948e9b0336274005b11939b64be527f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } health_report { overall_health: UNKNOWN } } }
I20260927 21:05:10.099064 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.096462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.16.190.195:33373)
0927 21:05:10.096531 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:10.096533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:10.096543 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:10.096592 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:10.096593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:10.096611 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:10.096698 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:10.096924 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.097405 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.098273 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:10.098465 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:10.098536 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:10.098718 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:10.098851 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:10.098939 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:10.098979 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 21:05:10.098999 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:05:10.101147 860 raft_consensus.cc:1261] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Refusing update from remote peer 948e9b0336274005b11939b64be527f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:10.101141 721 raft_consensus.cc:1261] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Refusing update from remote peer 948e9b0336274005b11939b64be527f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:10.101521 1054 consensus_queue.cc:1035] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:10.101660 1056 consensus_queue.cc:1035] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:10.104497 1060 mvcc.cc:205] Tried to move back new op lower bound from 7334064578971226112 to 7334064578910609408. Current Snapshot: MvccSnapshot[applied={T|T < 7334064578971226112 or (T in {7334064578971226112})}]
I20260927 21:05:10.107197 577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:10.107343 577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:10.108134 577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Sending DeleteTablet for 3 replicas of tablet dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:10.108429 972 tablet_service.cc:1514] Processing DeleteTablet for tablet dbab9dd64c64410dafe8e747fff82a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:10 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56504
I20260927 21:05:10.108477 701 tablet_service.cc:1514] Processing DeleteTablet for tablet dbab9dd64c64410dafe8e747fff82a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:10 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:56168
I20260927 21:05:10.108500 840 tablet_service.cc:1514] Processing DeleteTablet for tablet dbab9dd64c64410dafe8e747fff82a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:10 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:55494
I20260927 21:05:10.108564 776 tablet_replica.cc:330] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: stopping tablet replica
I20260927 21:05:10.108614 776 raft_consensus.cc:2229] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:10.108680 776 raft_consensus.cc:2258] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:10.108887 1070 tablet_replica.cc:330] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: stopping tablet replica
I20260927 21:05:10.108925 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 617
I20260927 21:05:10.109010 1070 raft_consensus.cc:2229] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:10.109112 1070 raft_consensus.cc:2258] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:10.109573 1070 ts_tablet_manager.cc:1905] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:10.110314 1069 tablet_replica.cc:330] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: stopping tablet replica
I20260927 21:05:10.110411 1069 raft_consensus.cc:2229] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:10.110534 1069 raft_consensus.cc:2258] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:10.111171 1069 ts_tablet_manager.cc:1905] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:10.111410 1070 ts_tablet_manager.cc:1918] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:10.111490 1070 log.cc:1196] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:10.111792 1070 ts_tablet_manager.cc:1939] T dbab9dd64c64410dafe8e747fff82a6d P 9b9b638400ca444aa7a5d8424fb9b50e: Deleting consensus metadata
I20260927 21:05:10.112311 564 catalog_manager.cc:4841] TS 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): tablet dbab9dd64c64410dafe8e747fff82a6d (table test-table [id=7e57b8eddca14d84aa3d70c672617f0b]) successfully deleted
I20260927 21:05:10.113042 1069 ts_tablet_manager.cc:1918] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:10.113119 1069 log.cc:1196] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:10.113422 1069 ts_tablet_manager.cc:1939] T dbab9dd64c64410dafe8e747fff82a6d P 948e9b0336274005b11939b64be527f5: Deleting consensus metadata
I20260927 21:05:10.113862 565 catalog_manager.cc:4841] TS 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): tablet dbab9dd64c64410dafe8e747fff82a6d (table test-table [id=7e57b8eddca14d84aa3d70c672617f0b]) successfully deleted
W20260927 21:05:10.115595 565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:46487 (error 108)
I20260927 21:05:10.115991 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 773
I20260927 21:05:10.116127 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:10.115982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:46487 (local address 127.0.0.1:56172)
0927 21:05:10.116049 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:10.116088 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:46487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 21:05:10.117043 565 catalog_manager.cc:4584] TS 5e36dfaf2458497384607f0e82c58f6a (127.16.190.193:46487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbab9dd64c64410dafe8e747fff82a6d: Network error: Client connection negotiation failed: client connection to 127.16.190.193:46487: connect: Connection refused (error 111)
I20260927 21:05:10.123112 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 909
I20260927 21:05:10.127832 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 546
I20260927 21:05:30.133249 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--webserver_interface=127.16.190.254
--webserver_port=45365
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:30.215399 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:30.215633 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:30.215657 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:30.216764 1072 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 21:05:30.216799 1072 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 21:05:30.216831 1072 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 21:05:30.216843 1072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:30.216856 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:30.216869 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:30.216881 1072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:30.216892 1072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:30.218336 1072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=45365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:30.218544 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:30.218721 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:30.224473 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:30.224507 1072 server_base.cc:1034] running on GCE node
W20260927 21:05:30.224476 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.224494 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:30.225051 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:30.225358 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:30.226521 1072 hybrid_clock.cc:648] HybridClock initialized: now 1790543130226504 us; error 30 us; skew 500 ppm
Sep 27 21:05:30 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:30.229295 1072 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:05:30.229833 1072 webserver.cc:460] Webserver started at http://127.16.190.254:45365/ using document root <none> and password file <none>
I20260927 21:05:30.230072 1072 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:30.230145 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:30.231598 1072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:30.232303 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.232487 1072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:30.232599 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a6229a26c9914480bc03c6a5d0a6dec9"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cbe1ad79ab065ab6971dae9c2d1498ac"
server_key_iv: "ddaa78d6d6cdcecb7aeb1276c2901baf"
server_key_version: "encryptionkey@0"
I20260927 21:05:30.233059 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:30.260143 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:30.262660 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:30.262843 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:30.267136 1072 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36853
I20260927 21:05:30.267203 1139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36853 every 8 connection(s)
I20260927 21:05:30.267609 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:30.269044 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1072
I20260927 21:05:30.269233 1140 sys_catalog.cc:263] Verifying existing consensus state
I20260927 21:05:30.269385 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:46487
--local_ip_for_outbound_sockets=127.16.190.193
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=33263
--webserver_interface=127.16.190.193
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:05:30.270052 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap starting.
I20260927 21:05:30.279098 1140 log.cc:824] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:30.281527 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:05:30.281859 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: Bootstrap complete.
I20260927 21:05:30.284116 1140 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:30.284384 1140 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Initialized, Role: FOLLOWER
I20260927 21:05:30.284598 1140 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:30.284698 1140 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:30.284754 1140 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:30.284804 1140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 2 FOLLOWER]: Advancing to term 3
I20260927 21:05:30.285867 1140 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:30.285995 1140 leader_election.cc:302] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6229a26c9914480bc03c6a5d0a6dec9; no voters:
I20260927 21:05:30.286209 1140 leader_election.cc:288] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 21:05:30.286247 1144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 21:05:30.286365 1144 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [term 3 LEADER]: Becoming Leader. State: Replica: a6229a26c9914480bc03c6a5d0a6dec9, State: Running, Role: LEADER
I20260927 21:05:30.286556 1140 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:30.286525 1144 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } }
I20260927 21:05:30.286976 1145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:05:30.287066 1145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:30.287366 1146 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6229a26c9914480bc03c6a5d0a6dec9. Latest consensus state: current_term: 3 leader_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6229a26c9914480bc03c6a5d0a6dec9" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36853 } } }
I20260927 21:05:30.287431 1146 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:30.288451 1160 catalog_manager.cc:1259] Loaded cluster ID: b77672ebc59743ae92aeb3bfeb774ac3
I20260927 21:05:30.288499 1160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: loading cluster ID for follower catalog manager: success
I20260927 21:05:30.289665 1160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6229a26c9914480bc03c6a5d0a6dec9: acquiring CA information for follower catalog manager: success
F20260927 21:05:30.290061 1160 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff71fb2e9ad google::LogMessage::Fail() at ??:0
@ 0x7ff71fb3188a google::LogMessage::SendToLog() at ??:0
@ 0x7ff71fb2e4de google::LogMessage::Flush() at ??:0
@ 0x7ff71fb2fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff72229b104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff727a876b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7ff727a87989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7ff727a87e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7ff727a8a8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7ff7219f1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff7228556db start_thread at ??:0
@ 0x7ff71e62c61f clone at ??:0
W20260927 21:05:30.367769 1142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:30.367990 1142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:30.368017 1142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:30.369334 1142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:30.369383 1142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:30.369419 1142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:30.370950 1142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:46487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=33263
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:30.371210 1142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:30.371474 1142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:30.376695 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.376737 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.376767 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:30.377084 1142 server_base.cc:1034] running on GCE node
I20260927 21:05:30.377343 1142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:30.377630 1142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:30.378813 1142 hybrid_clock.cc:648] HybridClock initialized: now 1790543130378793 us; error 36 us; skew 500 ppm
Sep 27 21:05:30 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:30.381549 1142 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:05:30.381999 1142 webserver.cc:460] Webserver started at http://127.16.190.193:33263/ using document root <none> and password file <none>
I20260927 21:05:30.382231 1142 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:30.382301 1142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:30.383700 1142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:05:30.384452 1188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.384660 1142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:30.384756 1142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "5e36dfaf2458497384607f0e82c58f6a"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "9c7f55458ba88fc073a22a1414f5974f"
server_key_iv: "b41fcff6fc2a37290b635387ff4e0902"
server_key_version: "encryptionkey@0"
I20260927 21:05:30.385183 1142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:30.398518 1142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:30.401243 1142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:30.401476 1142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:30.401736 1142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:30.402216 1195 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 21:05:30.404325 1142 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 21:05:30.404397 1142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 21:05:30.404459 1142 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 21:05:30.405201 1142 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 21:05:30.405236 1142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 21:05:30.405387 1195 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap starting.
I20260927 21:05:30.412726 1142 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:46487
I20260927 21:05:30.413159 1142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:30.414675 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1142
I20260927 21:05:30.414983 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:39713
--local_ip_for_outbound_sockets=127.16.190.194
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=41439
--webserver_interface=127.16.190.194
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:05:30.416980 1195 log.cc:824] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:30.419291 1195 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 21:05:30.419588 1195 tablet_bootstrap.cc:492] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Bootstrap complete.
I20260927 21:05:30.420513 1195 ts_tablet_manager.cc:1397] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260927 21:05:30.421660 1195 raft_consensus.cc:348] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:30.421923 1195 raft_consensus.cc:729] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e36dfaf2458497384607f0e82c58f6a, State: Initialized, Role: FOLLOWER
I20260927 21:05:30.422073 1195 consensus_queue.cc:260] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:30.422254 1302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:46487 every 8 connection(s)
I20260927 21:05:30.422385 1195 ts_tablet_manager.cc:1428] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260927 21:05:30.497958 1305 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:05:30.413899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:53031)
0927 21:05:30.414085 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.414130 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:30.414746 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:30.497726 (+ 82980us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
W20260927 21:05:30.498241 1303 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36853 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36853: Client connection negotiation failed: client connection to 127.16.190.254:36853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:05:30.498582 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.498458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:38099)
0927 21:05:30.498516 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.498554 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 21:05:30.498986 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.498895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:39899)
0927 21:05:30.498948 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.498972 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 21:05:30.499121 1303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 21:05:30.512930 1306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:30.513173 1306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:30.513212 1306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:30.514400 1306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:30.514475 1306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:30.514526 1306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:30.516001 1306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:39713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=41439
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:30.516283 1306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:30.516523 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:05:30.522545 1306 server_base.cc:1034] running on GCE node
W20260927 21:05:30.522481 1314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.522500 1316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.522689 1313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:30.523056 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:30.523319 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:30.524485 1306 hybrid_clock.cc:648] HybridClock initialized: now 1790543130524465 us; error 33 us; skew 500 ppm
Sep 27 21:05:30 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:30.526968 1306 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:05:30.527545 1306 webserver.cc:460] Webserver started at http://127.16.190.194:41439/ using document root <none> and password file <none>
I20260927 21:05:30.527791 1306 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:30.527838 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:30.529285 1306 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:30.529845 1323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.530016 1306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:05:30.530095 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "9b9b638400ca444aa7a5d8424fb9b50e"
format_stamp: "Formatted at 2026-09-27 21:04:48 on dist-test-slave-6c7s"
server_key: "cd7cb14d2b9f008149f1ae4c24d0db1c"
server_key_iv: "8822e34c5986afc172b0d60421d34286"
server_key_version: "encryptionkey@0"
I20260927 21:05:30.530426 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:30.544086 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:30.546613 1306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:30.546736 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:30.546944 1306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:30.547214 1306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:30.547245 1306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.547267 1306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:30.547279 1306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.553799 1306 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:39713
I20260927 21:05:30.553874 1436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:39713 every 8 connection(s)
I20260927 21:05:30.554226 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:30.555310 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.554939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:59165)
0927 21:05:30.555106 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.555175 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260927 21:05:30.555730 1437 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36853 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36853: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
I20260927 21:05:30.555999 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.555908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:45315)
0927 21:05:30.555957 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.555982 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 21:05:30.556224 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.556176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:46371)
0927 21:05:30.556196 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.556213 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 21:05:30.556326 1437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:05:30.561120 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1306
I20260927 21:05:30.561414 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:33373
--local_ip_for_outbound_sockets=127.16.190.195
--tserver_master_addrs=127.16.190.254:36853
--webserver_port=38871
--webserver_interface=127.16.190.195
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:30.643638 1440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:30.643863 1440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:30.643887 1440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:30.645128 1440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:30.645192 1440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:30.645227 1440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:30.646746 1440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:33373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=38871
--tserver_master_addrs=127.16.190.254:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:30.647012 1440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:30.647272 1440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:05:30.652529 1440 server_base.cc:1034] running on GCE node
W20260927 21:05:30.652500 1446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.652566 1445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:30.652500 1448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:30.653005 1440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:30.653277 1440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:30.654500 1440 hybrid_clock.cc:648] HybridClock initialized: now 1790543130654479 us; error 28 us; skew 500 ppm
Sep 27 21:05:30 dist-test-slave-6c7s krb5kdc[32465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:30.656980 1440 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:05:30.657526 1440 webserver.cc:460] Webserver started at http://127.16.190.195:38871/ using document root <none> and password file <none>
I20260927 21:05:30.657774 1440 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:30.657857 1440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:30.659190 1440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:30.659838 1455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.660009 1440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:30.660107 1440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "948e9b0336274005b11939b64be527f5"
format_stamp: "Formatted at 2026-09-27 21:04:49 on dist-test-slave-6c7s"
server_key: "4fd65cdca01868d24f4477b1be4f4a78"
server_key_iv: "52edf3e871c0a69c4e945cd494fdd3b8"
server_key_version: "encryptionkey@0"
I20260927 21:05:30.660521 1440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:30.673208 1440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:30.675514 1440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:30.675693 1440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:30.675951 1440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:30.676338 1440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:30.676401 1440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.676437 1440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:30.676465 1440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:30.682514 1440 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:33373
I20260927 21:05:30.682590 1569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:33373 every 8 connection(s)
I20260927 21:05:30.683007 1440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:30.684132 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.683734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:40237)
0927 21:05:30.683885 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.683983 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260927 21:05:30.684594 1570 heartbeater.cc:644] Failed to heartbeat to 127.16.190.254:36853 (0 consecutive failures): Network error: Failed to ping master at 127.16.190.254:36853: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
I20260927 21:05:30.684908 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.684804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:36115)
0927 21:05:30.684866 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.684892 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:05:30.685209 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:30.685160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:59947)
0927 21:05:30.685180 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:30.685197 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 21:05:30.685320 1570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 21:05:30.686389 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1440
I20260927 21:05:31.499797 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.499548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:55601)
0927 21:05:31.499680 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:31.499728 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260927 21:05:31.557163 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.556706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:46997)
0927 21:05:31.557022 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:31.557080 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":139,"threads_started":1}
I20260927 21:05:31.685900 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.685662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:42967)
0927 21:05:31.685801 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:31.685841 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260927 21:05:31.862356 1578 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:31.862561 1578 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:31.863024 1578 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
Sep 27 21:05:31 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.195@KRBTEST.COM
Sep 27 21:05:31 dist-test-slave-6c7s krb5kdc[32465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543130, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.194@KRBTEST.COM
I20260927 21:05:31.876888 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.863336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:35881 (local address 127.16.190.194:39713)
0927 21:05:31.863503 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:31.863515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:31.863551 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:05:31.864160 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:31.864163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:31.864270 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:31.864383 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:31.865471 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.866881 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.868373 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.869043 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.869160 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:31.872789 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:31.872797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:31.872798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:31.872808 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:31.874869 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875462 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:31.875466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:31.875467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:31.875601 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875933 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:31.875934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:31.875934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:31.876197 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:31.876303 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:31.876611 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:05:31.876744 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260927 21:05:31.876886 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.863245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:33373 (local address 127.16.190.193:48831)
0927 21:05:31.863306 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:31.863317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:31.863378 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:31.864468 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:31.864471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:31.865031 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:31.865322 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:31.865330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.867016 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.867017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:31.868272 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:31.868275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.869150 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.869152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:31.869317 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:31.870056 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:31.870068 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:31.872248 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:31.874905 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:31.874912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:31.874915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:31.875379 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:31.875731 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:31.875732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:31.875857 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:31.876288 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:31.876296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:31.876432 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 21:05:31.876556 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:05:31.876757 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":62}
I20260927 21:05:31.876941 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.863377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:48831 (local address 127.16.190.195:33373)
0927 21:05:31.863561 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:31.863569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:31.863593 (+ 24us) server_negotiation.cc:407] Connection header received
0927 21:05:31.864157 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:31.864160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:31.864220 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:31.864384 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:31.865471 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.866881 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.868373 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.869043 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.869160 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:31.872428 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:31.872433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:31.872433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:31.872450 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:31.874788 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875522 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:31.875526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:31.875526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:31.875643 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875950 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:31.875950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:31.875951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:31.876197 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:31.876305 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:31.876611 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 21:05:31.876782 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
I20260927 21:05:31.877018 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:31.863279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:39713 (local address 127.16.190.193:35881)
0927 21:05:31.863399 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:31.863410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:31.863465 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:31.864468 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:31.864471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:31.865032 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:31.865322 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:31.865330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.867007 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.867009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:31.868272 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:31.868275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:31.869152 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:31.869153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:31.869317 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:31.870056 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:31.870068 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:31.872630 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:31.874955 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:31.874955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:31.874956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:31.875379 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:31.875698 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:31.875698 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:31.875699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:31.875857 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:31.876285 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:31.876292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:31.876432 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 21:05:31.876556 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 21:05:31.876876 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":219,"thread_start_us":58,"threads_started":1}
W20260927 21:05:31.878082 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:31.878270 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:31.878319 1191 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:31.878417 1578 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:32.500586 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:32.500385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:38397)
0927 21:05:32.500509 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:32.500539 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260927 21:05:32.558096 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:32.557824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:48423)
0927 21:05:32.558005 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:32.558051 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260927 21:05:32.686650 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:32.686446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:38263)
0927 21:05:32.686566 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:32.686608 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 21:05:33.419003 1585 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:33.419099 1585 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:33.419271 1585 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
W20260927 21:05:33.419701 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:33.419770 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:33.419797 1190 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:33.419903 1585 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:33.501279 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:33.501028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:49701)
0927 21:05:33.501196 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:33.501232 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260927 21:05:33.558961 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:33.558734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:57455)
0927 21:05:33.558862 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:33.558913 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260927 21:05:33.687404 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:33.687121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:58471)
0927 21:05:33.687321 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:33.687362 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":61,"threads_started":1}
I20260927 21:05:34.502131 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:34.501861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:49051)
0927 21:05:34.502036 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:34.502082 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260927 21:05:34.559784 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:34.559526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:50327)
0927 21:05:34.559692 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:34.559736 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260927 21:05:34.688270 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:34.688041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:41593)
0927 21:05:34.688166 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:34.688215 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260927 21:05:35.241431 1592 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:35.241576 1592 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:35.241787 1592 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
W20260927 21:05:35.242177 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:35.242254 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:35.242336 1191 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:35.242408 1592 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:35.502990 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:35.502738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:34513)
0927 21:05:35.502902 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:35.502941 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260927 21:05:35.560590 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:35.560350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:50751)
0927 21:05:35.560496 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:35.560540 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260927 21:05:35.689116 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:35.688876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:44077)
0927 21:05:35.689028 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:35.689073 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260927 21:05:36.503804 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:36.503534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:60569)
0927 21:05:36.503702 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:36.503753 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260927 21:05:36.561326 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:36.561081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:37095)
0927 21:05:36.561239 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:36.561279 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260927 21:05:36.689896 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:36.689609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:37379)
0927 21:05:36.689800 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:36.689850 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260927 21:05:37.504691 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:37.504433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:36093)
0927 21:05:37.504591 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:37.504639 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260927 21:05:37.562093 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:37.561826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:37677)
0927 21:05:37.561999 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:37.562043 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260927 21:05:37.690622 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:37.690383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:47865)
0927 21:05:37.690537 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:37.690577 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260927 21:05:38.505702 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:38.505392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:33501)
0927 21:05:38.505604 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:38.505646 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260927 21:05:38.563037 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:38.562793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:45895)
0927 21:05:38.562945 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:38.562990 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260927 21:05:38.691395 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:38.691176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:56081)
0927 21:05:38.691305 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:38.691345 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260927 21:05:39.506425 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:39.506214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:60581)
0927 21:05:39.506333 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:39.506378 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260927 21:05:39.564047 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:39.563687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:35393)
0927 21:05:39.563904 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:39.563991 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260927 21:05:39.692301 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:39.691996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:42989)
0927 21:05:39.692158 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:39.692249 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260927 21:05:39.830330 1608 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:39.830474 1608 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:39.830696 1608 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
W20260927 21:05:39.831199 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:39.831324 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:39.831391 1190 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:39.831461 1608 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:40.507405 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:40.507146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:34331)
0927 21:05:40.507303 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:40.507349 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260927 21:05:40.564970 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:40.564632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:43209)
0927 21:05:40.564808 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:40.564923 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260927 21:05:40.693246 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:40.692990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:46773)
0927 21:05:40.693150 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:40.693196 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260927 21:05:41.508285 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:41.508034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:52475)
0927 21:05:41.508189 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:41.508237 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260927 21:05:41.565842 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:41.565599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:45419)
0927 21:05:41.565751 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:41.565797 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260927 21:05:41.694070 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:41.693825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:52709)
0927 21:05:41.693978 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:41.694023 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260927 21:05:41.773017 1615 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:41.773113 1615 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:41.773322 1615 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
W20260927 21:05:41.773788 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:41.773891 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:41.773967 1190 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:41.774057 1615 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:42.509104 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:42.508887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:55397)
0927 21:05:42.509004 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:42.509046 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260927 21:05:42.566669 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:42.566409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:41907)
0927 21:05:42.566578 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:42.566620 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 21:05:42.694850 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:42.694614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:55843)
0927 21:05:42.694760 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:42.694800 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260927 21:05:43.509891 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:43.509645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:59971)
0927 21:05:43.509785 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:43.509841 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260927 21:05:43.567577 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:43.567187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:38129)
0927 21:05:43.567460 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:43.567510 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":161,"threads_started":1}
I20260927 21:05:43.695605 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:43.695362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:56307)
0927 21:05:43.695518 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:43.695559 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260927 21:05:44.058039 1622 raft_consensus.cc:482] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:44.058144 1622 raft_consensus.cc:504] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9b638400ca444aa7a5d8424fb9b50e" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 39713 } } peers { permanent_uuid: "948e9b0336274005b11939b64be527f5" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 33373 } } peers { permanent_uuid: "5e36dfaf2458497384607f0e82c58f6a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 46487 } }
I20260927 21:05:44.058396 1622 leader_election.cc:288] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713), 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373)
W20260927 21:05:44.058836 1191 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 948e9b0336274005b11939b64be527f5 (127.16.190.195:33373): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
W20260927 21:05:44.059018 1190 leader_election.cc:341] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9b638400ca444aa7a5d8424fb9b50e (127.16.190.194:39713): Not found: Tablet not found: dbab9dd64c64410dafe8e747fff82a6d
I20260927 21:05:44.059089 1190 leader_election.cc:302] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e36dfaf2458497384607f0e82c58f6a; no voters: 948e9b0336274005b11939b64be527f5, 9b9b638400ca444aa7a5d8424fb9b50e
I20260927 21:05:44.059168 1622 raft_consensus.cc:2726] T dbab9dd64c64410dafe8e747fff82a6d P 5e36dfaf2458497384607f0e82c58f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 21:05:44.510752 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:44.510485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:45121)
0927 21:05:44.510649 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:44.510700 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260927 21:05:44.568496 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:44.568170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:56457)
0927 21:05:44.568404 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:44.568448 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260927 21:05:44.696440 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:44.696179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:36301)
0927 21:05:44.696347 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:44.696389 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260927 21:05:45.511832 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:45.511581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.193:42881)
0927 21:05:45.511729 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:45.511772 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260927 21:05:45.569370 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:45.569131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.194:41117)
0927 21:05:45.569275 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:45.569322 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260927 21:05:45.692543 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1142
I20260927 21:05:45.697342 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:45.697093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:36853 (local address 127.16.190.195:36609)
0927 21:05:45.697257 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:45.697288 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.254:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260927 21:05:45.697510 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1306
I20260927 21:05:45.701943 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1440
2026-09-27T21:05:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58435 ms)
[----------] 30 tests from SecurityITest (238866 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:45 dist-test-slave-6c7s krb5kdc[1631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:45 dist-test-slave-6c7s krb5kdc[1631](info): set up 2 sockets
Sep 27 21:05:45 dist-test-slave-6c7s krb5kdc[1631](info): commencing operation
krb5kdc: starting...
W20260927 21:05:46.753407 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:46 dist-test-slave-6c7s krb5kdc[1631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:05:46.881139 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39817
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:46.966895 1644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:46.967185 1644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:46.967226 1644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:46.968495 1644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:46.968580 1644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:46.968603 1644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:46.968624 1644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:46.968652 1644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:46.970297 1644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:46.970616 1644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:46.970903 1644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:46.978116 1651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:46.978135 1650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:46.978137 1644 server_base.cc:1034] running on GCE node
W20260927 21:05:46.978132 1653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:46.978727 1644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:46.979044 1644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:46.980285 1644 hybrid_clock.cc:648] HybridClock initialized: now 1790543146980206 us; error 68 us; skew 500 ppm
Sep 27 21:05:46 dist-test-slave-6c7s krb5kdc[1631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:46.983150 1644 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:05:46.983785 1644 webserver.cc:460] Webserver started at http://127.16.190.254:39301/ using document root <none> and password file <none>
I20260927 21:05:46.984033 1644 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:46.984105 1644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:46.984277 1644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:46.985502 1644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "8ff04e5da89a499daa902a810b05acc3"
format_stamp: "Formatted at 2026-09-27 21:05:46 on dist-test-slave-6c7s"
server_key: "72e9a1c1d6065266dec26278c73027f8"
server_key_iv: "e76bb5c09d7b6635180abdf4996d4c33"
server_key_version: "encryptionkey@0"
I20260927 21:05:46.985903 1644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "8ff04e5da89a499daa902a810b05acc3"
format_stamp: "Formatted at 2026-09-27 21:05:46 on dist-test-slave-6c7s"
server_key: "72e9a1c1d6065266dec26278c73027f8"
server_key_iv: "e76bb5c09d7b6635180abdf4996d4c33"
server_key_version: "encryptionkey@0"
I20260927 21:05:46.987417 1644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:05:46.988394 1660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:46.988869 1644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:46.989013 1644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "8ff04e5da89a499daa902a810b05acc3"
format_stamp: "Formatted at 2026-09-27 21:05:46 on dist-test-slave-6c7s"
server_key: "72e9a1c1d6065266dec26278c73027f8"
server_key_iv: "e76bb5c09d7b6635180abdf4996d4c33"
server_key_version: "encryptionkey@0"
I20260927 21:05:46.989107 1644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:47.020321 1644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:47.022938 1644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:47.023130 1644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:47.027403 1644 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39817
I20260927 21:05:47.027449 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39817 every 8 connection(s)
I20260927 21:05:47.027825 1644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:47.028875 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:47.031409 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: Bootstrap starting.
I20260927 21:05:47.032070 1713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:47.032410 1713 log.cc:824] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:47.033466 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: No bootstrap required, opened a new log
I20260927 21:05:47.035378 1713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } }
I20260927 21:05:47.035545 1713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:47.035570 1713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff04e5da89a499daa902a810b05acc3, State: Initialized, Role: FOLLOWER
I20260927 21:05:47.035753 1713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } }
I20260927 21:05:47.035847 1713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:47.035873 1713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:47.035930 1713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:47.036581 1713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } }
I20260927 21:05:47.036679 1713 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff04e5da89a499daa902a810b05acc3; no voters:
I20260927 21:05:47.036923 1713 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:47.037129 1716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:47.037309 1713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:47.037309 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1644
I20260927 21:05:47.037353 1716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff04e5da89a499daa902a810b05acc3, State: Running, Role: LEADER
I20260927 21:05:47.037424 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:47.037542 1716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } }
I20260927 21:05:47.037606 17147 external_mini_cluster.cc:1428] Setting key 58c38bebfc2c784cf4e84852ed1a0dd2
I20260927 21:05:47.038177 1718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff04e5da89a499daa902a810b05acc3. Latest consensus state: current_term: 1 leader_uuid: "8ff04e5da89a499daa902a810b05acc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } } }
I20260927 21:05:47.038375 1718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:47.038592 1717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff04e5da89a499daa902a810b05acc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff04e5da89a499daa902a810b05acc3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39817 } } }
I20260927 21:05:47.038880 1717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:47.039134 1728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:47.039784 1728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:47.041590 1728 catalog_manager.cc:1347] Generated new cluster ID: 1c9873bb99b0479c8bd81990ec89076d
I20260927 21:05:47.041654 1728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:47.049630 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.038417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.16.190.254:39817)
0927 21:05:47.038787 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.038798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.038838 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:05:47.038973 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.038981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.039057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.039925 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.040030 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.041261 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.042226 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.042672 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.042797 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.045363 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.045368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.045369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:47.045384 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.047815 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.048237 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.048241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.048241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.048366 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.048587 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.048588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.048588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.048941 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.049172 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.049247 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.049410 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":81,"threads_started":1}
I20260927 21:05:47.067101 1728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:47.067829 1728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:47.078924 1728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: Generated new TSK 0
I20260927 21:05:47.079247 1728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:05:47.116916 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:47.209019 1739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:47.209246 1739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:47.209271 1739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:47.210498 1739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:47.210556 1739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:47.210584 1739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:47.212230 1739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:47.212517 1739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:47.212764 1739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:47.218318 1747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.218389 1745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.218379 1744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.218559 1739 server_base.cc:1034] running on GCE node
I20260927 21:05:47.218807 1739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:47.219097 1739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:47.220299 1739 hybrid_clock.cc:648] HybridClock initialized: now 1790543147220269 us; error 50 us; skew 500 ppm
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:47.223158 1739 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:05:47.223649 1739 webserver.cc:460] Webserver started at http://127.16.190.193:34273/ using document root <none> and password file <none>
I20260927 21:05:47.223913 1739 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:47.223984 1739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:47.224133 1739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:47.225420 1739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "673d0be286424eafa837a852f9cb5ec7"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "8b2ebec4abab6439c572f3ea2afe8991"
server_key_iv: "4addf536b3a677e62f2d9960db49ea71"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.225829 1739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "673d0be286424eafa837a852f9cb5ec7"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "8b2ebec4abab6439c572f3ea2afe8991"
server_key_iv: "4addf536b3a677e62f2d9960db49ea71"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.227304 1739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:05:47.228191 1754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.228394 1739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:47.228500 1739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "673d0be286424eafa837a852f9cb5ec7"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "8b2ebec4abab6439c572f3ea2afe8991"
server_key_iv: "4addf536b3a677e62f2d9960db49ea71"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.228581 1739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:47.251921 1739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:47.254297 1739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:47.254487 1739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:47.254745 1739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:47.255090 1739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:47.255147 1739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.255189 1739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:47.255219 1739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.261611 1739 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:37793
I20260927 21:05:47.261657 1868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:37793 every 8 connection(s)
I20260927 21:05:47.262005 1739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:47.263015 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1739
I20260927 21:05:47.263139 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:47.263352 17147 external_mini_cluster.cc:1428] Setting key a10494ee81814e13ef58d9c000d4a3bb
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:05:47.274199 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.262814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:58395 (local address 127.16.190.254:39817)
0927 21:05:47.262885 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.262887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.263109 (+ 222us) server_negotiation.cc:407] Connection header received
0927 21:05:47.263855 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.263856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.263870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.263921 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.265032 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.265727 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.267452 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.267702 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.267792 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.270676 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.270678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.270679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:47.270688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.272363 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.272880 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.272882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.272882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.272956 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.273419 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.273420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.273421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.273547 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.273614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.273996 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.274101 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:05:47.274401 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.262786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39817 (local address 127.16.190.193:58395)
0927 21:05:47.262973 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.263013 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.263743 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.264013 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.264018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.264676 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:47.264927 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.264932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.265839 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.265840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.267331 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.267335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.267820 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.267821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.268014 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.268606 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:47.268619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:47.270537 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:47.272457 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.272466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.272468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.272697 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.273130 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.273132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.273133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.273297 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.273635 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:47.273642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:47.273786 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.273964 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.274143 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260927 21:05:47.275158 1869 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39817
I20260927 21:05:47.275264 1869 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:47.275485 1869 heartbeater.cc:511] Master 127.16.190.254:39817 requested a full tablet report, sending...
I20260927 21:05:47.276093 1677 ts_manager.cc:194] Registered new tserver with Master: 673d0be286424eafa837a852f9cb5ec7 (127.16.190.193:37793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:05:47.276991 1677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:58395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:05:47.293337 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:47.381505 1874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:47.381796 1874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:47.381842 1874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:47.383124 1874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:47.383198 1874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:47.383245 1874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:47.384912 1874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:47.385186 1874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:47.385439 1874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:47.391459 1880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.391661 1874 server_base.cc:1034] running on GCE node
W20260927 21:05:47.391407 1882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.391407 1879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.392004 1874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:47.392302 1874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:47.393484 1874 hybrid_clock.cc:648] HybridClock initialized: now 1790543147393496 us; error 58 us; skew 500 ppm
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:47.396351 1874 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:05:47.396940 1874 webserver.cc:460] Webserver started at http://127.16.190.194:41323/ using document root <none> and password file <none>
I20260927 21:05:47.397217 1874 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:47.397298 1874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:47.397454 1874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:47.398671 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "fd82b6531a1b478e96c266d79acc5b6a"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "d7684831df8a34c484c088156629a9be"
server_key_iv: "eb9f18f1a8f5c22c8f5c93cac99e844c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.399065 1874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "fd82b6531a1b478e96c266d79acc5b6a"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "d7684831df8a34c484c088156629a9be"
server_key_iv: "eb9f18f1a8f5c22c8f5c93cac99e844c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.400605 1874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:47.401494 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.401727 1874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:47.401834 1874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "fd82b6531a1b478e96c266d79acc5b6a"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "d7684831df8a34c484c088156629a9be"
server_key_iv: "eb9f18f1a8f5c22c8f5c93cac99e844c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.401921 1874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:47.410667 1874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:47.413303 1874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:47.413493 1874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:47.413803 1874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:47.414135 1874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:47.414191 1874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.414232 1874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:47.414278 1874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.421499 1874 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44151
I20260927 21:05:47.421561 2003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44151 every 8 connection(s)
I20260927 21:05:47.421954 1874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:47.429010 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 1874
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:47.429124 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:47.429359 17147 external_mini_cluster.cc:1428] Setting key fd42621bf5a01eeeaeeaa23f4c038394
I20260927 21:05:47.433131 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.422810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:57015 (local address 127.16.190.254:39817)
0927 21:05:47.422865 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.422867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.423068 (+ 201us) server_negotiation.cc:407] Connection header received
0927 21:05:47.423723 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.423724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.423740 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.423797 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.424998 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.425573 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.426734 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.426972 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.427059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.429749 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.429750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.429751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:47.429758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.431492 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.432016 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.432018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.432018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.432083 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.432349 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.432350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.432350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.432475 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.432547 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.432944 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.433035 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:05:47.433341 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.422763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39817 (local address 127.16.190.194:57015)
0927 21:05:47.422920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.422971 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.423587 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.423882 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.423887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.424561 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:47.424797 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.424803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.425698 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.425699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.426629 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.426631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.427080 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.427082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.427232 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.427879 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:47.427889 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:47.429587 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:47.431637 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.431645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.431648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.431927 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.432157 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.432158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.432158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.432279 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.432549 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:47.432556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:47.432776 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.432923 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.433098 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 21:05:47.433794 2004 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39817
I20260927 21:05:47.433923 2004 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:47.434118 2004 heartbeater.cc:511] Master 127.16.190.254:39817 requested a full tablet report, sending...
I20260927 21:05:47.434545 1677 ts_manager.cc:194] Registered new tserver with Master: fd82b6531a1b478e96c266d79acc5b6a (127.16.190.194:44151)
I20260927 21:05:47.435040 1677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:57015
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:05:47.461476 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:47.549973 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:47.550197 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:47.550231 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:47.551399 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:47.551440 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:47.551467 2009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:47.553000 2009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:39817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:47.553210 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:47.553416 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:05:47.558661 2009 server_base.cc:1034] running on GCE node
W20260927 21:05:47.558600 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.558614 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.558619 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.559137 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:47.559486 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:47.560686 2009 hybrid_clock.cc:648] HybridClock initialized: now 1790543147560678 us; error 42 us; skew 500 ppm
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:47.563472 2009 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:05:47.563938 2009 webserver.cc:460] Webserver started at http://127.16.190.195:45789/ using document root <none> and password file <none>
I20260927 21:05:47.564193 2009 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:47.564262 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:47.564417 2009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:47.565593 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "78b52b29dd8c409f9fc19ac7297b0f61"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "3f0372f2f72ebf965a86f9e89746db88"
server_key_iv: "8bfc844e67d183c06cac8df853de77c3"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.565954 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "78b52b29dd8c409f9fc19ac7297b0f61"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "3f0372f2f72ebf965a86f9e89746db88"
server_key_iv: "8bfc844e67d183c06cac8df853de77c3"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.567485 2009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:47.568373 2024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.568602 2009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:47.568711 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "78b52b29dd8c409f9fc19ac7297b0f61"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "3f0372f2f72ebf965a86f9e89746db88"
server_key_iv: "8bfc844e67d183c06cac8df853de77c3"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.568794 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:47.580240 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:47.582845 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:47.583014 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:47.583266 2009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:47.583582 2009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:47.583616 2009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.583638 2009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:47.583686 2009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.589809 2009 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:42151
I20260927 21:05:47.589851 2138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:42151 every 8 connection(s)
I20260927 21:05:47.590219 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:47.597035 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2009
I20260927 21:05:47.597167 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
Sep 27 21:05:47 dist-test-slave-6c7s krb5kdc[1631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:47.597398 17147 external_mini_cluster.cc:1428] Setting key 152958d8dd0495bc70acd3c2bd6cf1a2
I20260927 21:05:47.601161 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.591127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:36097 (local address 127.16.190.254:39817)
0927 21:05:47.591191 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.591193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.591365 (+ 172us) server_negotiation.cc:407] Connection header received
0927 21:05:47.591856 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.591858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.591871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.591950 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.593208 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.593772 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.594907 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.595099 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.595169 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.597969 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.597970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.597970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:47.597978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.599764 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.600214 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.600214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.600215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.600272 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.600483 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.600484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.600484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.600596 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.600650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.601026 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.601093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:05:47.601424 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.591042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:39817 (local address 127.16.190.195:36097)
0927 21:05:47.591204 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.591247 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.591737 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.592054 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.592057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.592754 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:47.593038 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.593044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.593877 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.593878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.594789 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.594792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.595236 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.595238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.595353 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.595907 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:47.595915 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:47.597859 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:47.599869 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.599877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.599879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.600136 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.600318 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:47.600319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:47.600319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:47.600426 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:47.600679 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:47.600686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:47.600885 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.601010 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.601150 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260927 21:05:47.601987 2139 heartbeater.cc:347] Connected to a master server at 127.16.190.254:39817
I20260927 21:05:47.602080 2139 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:47.602286 2139 heartbeater.cc:511] Master 127.16.190.254:39817 requested a full tablet report, sending...
I20260927 21:05:47.602708 1677 ts_manager.cc:194] Registered new tserver with Master: 78b52b29dd8c409f9fc19ac7297b0f61 (127.16.190.195:42151)
I20260927 21:05:47.603094 1677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:36097
I20260927 21:05:47.609640 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:47.617545 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.611059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.16.190.254:39817)
0927 21:05:47.611157 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.611160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.611258 (+ 98us) server_negotiation.cc:407] Connection header received
0927 21:05:47.611344 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.611345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.611356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.611409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.612008 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.612495 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.613297 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.613496 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.613585 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.614870 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.614871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.614872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:47.614878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.616310 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.616684 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.616685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.616685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.616736 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:47.617003 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:47.617004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:47.617004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:47.617149 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.617235 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.617389 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.617451 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 21:05:47.619243 1677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:47.620877 1677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:47.628157 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.624014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.16.190.194:44151)
0927 21:05:47.624165 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.624171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.624206 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:05:47.624379 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.624384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.624401 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.624458 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:47.624758 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.625631 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.627048 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.627742 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.627842 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.627906 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.628019 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.628063 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260927 21:05:47.628160 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.623918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44151 (local address 127.0.0.1:50450)
0927 21:05:47.624103 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.624149 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.624231 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.624606 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.624610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.624612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:47.624687 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.624689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.625741 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.625742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.626954 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.626957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.627839 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.627840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.627921 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.627934 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.628005 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.628063 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260927 21:05:47.628386 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.624169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:37793 (local address 127.0.0.1:55080)
0927 21:05:47.624275 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.624285 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.624325 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.624860 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.624861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.624862 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:47.624899 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.624900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.626186 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.626186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.627188 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.627190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.628141 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.628142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.628207 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.628212 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.628244 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.628287 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260927 21:05:47.628577 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.623932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:42151 (local address 127.0.0.1:59832)
0927 21:05:47.624176 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.624187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.624241 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.625010 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.625011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.625013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:47.625056 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.625057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.626323 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.626324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.627383 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.627386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.628336 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.628336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.628392 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.628396 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.628462 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.628502 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":114,"threads_started":1}
I20260927 21:05:47.628667 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.624317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.16.190.195:42151)
0927 21:05:47.624487 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.624492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.624501 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:05:47.624543 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.624546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.624562 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.624640 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:47.625149 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.626206 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.627473 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.628188 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.628255 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.628303 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.628501 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.628559 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260927 21:05:47.629462 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.624493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.16.190.193:37793)
0927 21:05:47.624677 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.624681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.624691 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:47.624729 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.624731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.624747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.624858 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:47.624957 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.626070 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.627289 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.628035 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.629232 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.629299 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.629328 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.629375 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260927 21:05:47.629441 2072 tablet_service.cc:1467] Processing CreateTablet for tablet fe73816f00c842f2a0e7f5faba517c07 (DEFAULT_TABLE table=test-table [id=774ffe0649824246bb0ab9b1bb6ad3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:47.629480 1938 tablet_service.cc:1467] Processing CreateTablet for tablet fe73816f00c842f2a0e7f5faba517c07 (DEFAULT_TABLE table=test-table [id=774ffe0649824246bb0ab9b1bb6ad3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:47.629873 2072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe73816f00c842f2a0e7f5faba517c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:47.629930 1938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe73816f00c842f2a0e7f5faba517c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:47.630213 1803 tablet_service.cc:1467] Processing CreateTablet for tablet fe73816f00c842f2a0e7f5faba517c07 (DEFAULT_TABLE table=test-table [id=774ffe0649824246bb0ab9b1bb6ad3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:47.630527 1803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe73816f00c842f2a0e7f5faba517c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:47.632936 2156 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Bootstrap starting.
I20260927 21:05:47.633492 2157 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Bootstrap starting.
I20260927 21:05:47.633845 2158 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Bootstrap starting.
I20260927 21:05:47.634032 2156 tablet_bootstrap.cc:654] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:47.634143 2157 tablet_bootstrap.cc:654] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:47.634441 2158 tablet_bootstrap.cc:654] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:47.634709 2157 log.cc:824] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:47.634709 2156 log.cc:824] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:47.635041 2158 log.cc:824] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:47.635509 2157 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: No bootstrap required, opened a new log
I20260927 21:05:47.635586 2157 ts_tablet_manager.cc:1397] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:47.635690 2156 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: No bootstrap required, opened a new log
I20260927 21:05:47.635766 2156 ts_tablet_manager.cc:1397] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:05:47.635779 2158 tablet_bootstrap.cc:492] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: No bootstrap required, opened a new log
I20260927 21:05:47.635849 2158 ts_tablet_manager.cc:1397] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:47.637214 2157 raft_consensus.cc:348] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.637483 2157 raft_consensus.cc:374] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:47.637532 2157 raft_consensus.cc:729] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd82b6531a1b478e96c266d79acc5b6a, State: Initialized, Role: FOLLOWER
I20260927 21:05:47.637665 2157 consensus_queue.cc:260] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.637995 2157 ts_tablet_manager.cc:1428] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 21:05:47.637979 2156 raft_consensus.cc:348] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.637945 2158 raft_consensus.cc:348] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.638135 2158 raft_consensus.cc:374] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:47.638134 2156 raft_consensus.cc:374] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:47.637995 2004 heartbeater.cc:503] Master 127.16.190.254:39817 was elected leader, sending a full tablet report...
I20260927 21:05:47.638180 2158 raft_consensus.cc:729] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 673d0be286424eafa837a852f9cb5ec7, State: Initialized, Role: FOLLOWER
I20260927 21:05:47.638180 2156 raft_consensus.cc:729] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b52b29dd8c409f9fc19ac7297b0f61, State: Initialized, Role: FOLLOWER
I20260927 21:05:47.638293 2158 consensus_queue.cc:260] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.638365 2156 consensus_queue.cc:260] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.638605 2158 ts_tablet_manager.cc:1428] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:47.638645 2156 ts_tablet_manager.cc:1428] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:47.638725 1869 heartbeater.cc:503] Master 127.16.190.254:39817 was elected leader, sending a full tablet report...
I20260927 21:05:47.639012 2139 heartbeater.cc:503] Master 127.16.190.254:39817 was elected leader, sending a full tablet report...
W20260927 21:05:47.672649 2005 tablet.cc:2369] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 21:05:47.762921 1870 tablet.cc:2369] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:05:47.768043 2164 raft_consensus.cc:482] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:47.768169 2164 raft_consensus.cc:504] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.768607 2164 leader_election.cc:288] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd82b6531a1b478e96c266d79acc5b6a (127.16.190.194:44151), 673d0be286424eafa837a852f9cb5ec7 (127.16.190.193:37793)
I20260927 21:05:47.772840 2141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.768871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44151 (local address 127.16.190.195:37219)
0927 21:05:47.768930 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.768947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.769017 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.769246 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.769247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.769249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:47.769374 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.769375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.770382 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.770383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.771697 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.771700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.772516 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.772517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.772611 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.772613 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.772682 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.772721 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":80}
I20260927 21:05:47.772871 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.768872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:37793 (local address 127.16.190.195:55417)
0927 21:05:47.769064 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.769072 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:47.769135 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:47.769266 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:47.769267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:47.769268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:47.769341 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.769343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.770327 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.770328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.771697 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:47.771700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.772516 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.772517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:47.772601 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.772606 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:47.772671 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 21:05:47.772720 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":98,"thread_start_us":99,"threads_started":1}
I20260927 21:05:47.772876 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.768948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:55417 (local address 127.16.190.193:37793)
0927 21:05:47.768991 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.768994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.769155 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:05:47.769185 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.769186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.769199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.769253 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:47.769452 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.770227 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.771805 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.772427 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.772507 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.772517 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.772690 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.772739 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:05:47.772886 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.768991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:37219 (local address 127.16.190.194:44151)
0927 21:05:47.769044 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.769046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.769058 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:47.769095 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.769096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.769112 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.769200 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:47.769453 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.770219 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.771805 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.772431 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.772526 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.772535 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.772764 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.772824 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:05:47.773238 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe73816f00c842f2a0e7f5faba517c07" candidate_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd82b6531a1b478e96c266d79acc5b6a" is_pre_election: true
I20260927 21:05:47.773427 1958 raft_consensus.cc:2454] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b52b29dd8c409f9fc19ac7297b0f61 in term 0.
I20260927 21:05:47.773578 1823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe73816f00c842f2a0e7f5faba517c07" candidate_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673d0be286424eafa837a852f9cb5ec7" is_pre_election: true
I20260927 21:05:47.773729 2028 leader_election.cc:302] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b52b29dd8c409f9fc19ac7297b0f61, fd82b6531a1b478e96c266d79acc5b6a; no voters:
I20260927 21:05:47.773761 1823 raft_consensus.cc:2454] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b52b29dd8c409f9fc19ac7297b0f61 in term 0.
I20260927 21:05:47.773896 2164 raft_consensus.cc:2781] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:47.773960 2164 raft_consensus.cc:482] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:47.773999 2164 raft_consensus.cc:3037] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:47.774876 2164 raft_consensus.cc:504] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.775028 2164 leader_election.cc:288] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [CANDIDATE]: Term 1 election: Requested vote from peers fd82b6531a1b478e96c266d79acc5b6a (127.16.190.194:44151), 673d0be286424eafa837a852f9cb5ec7 (127.16.190.193:37793)
I20260927 21:05:47.775221 1823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe73816f00c842f2a0e7f5faba517c07" candidate_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "673d0be286424eafa837a852f9cb5ec7"
I20260927 21:05:47.775257 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe73816f00c842f2a0e7f5faba517c07" candidate_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd82b6531a1b478e96c266d79acc5b6a"
I20260927 21:05:47.775331 1823 raft_consensus.cc:3037] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:47.775384 1958 raft_consensus.cc:3037] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:47.776388 1958 raft_consensus.cc:2454] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b52b29dd8c409f9fc19ac7297b0f61 in term 1.
I20260927 21:05:47.776387 1823 raft_consensus.cc:2454] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b52b29dd8c409f9fc19ac7297b0f61 in term 1.
I20260927 21:05:47.776683 2028 leader_election.cc:302] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b52b29dd8c409f9fc19ac7297b0f61, fd82b6531a1b478e96c266d79acc5b6a; no voters:
I20260927 21:05:47.776876 2164 raft_consensus.cc:2781] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:47.777087 2164 raft_consensus.cc:686] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 1 LEADER]: Becoming Leader. State: Replica: 78b52b29dd8c409f9fc19ac7297b0f61, State: Running, Role: LEADER
I20260927 21:05:47.777225 2164 consensus_queue.cc:237] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } }
I20260927 21:05:47.778141 1676 catalog_manager.cc:5495] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b52b29dd8c409f9fc19ac7297b0f61 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b52b29dd8c409f9fc19ac7297b0f61" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 42151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 } health_report { overall_health: UNKNOWN } } }
I20260927 21:05:47.811614 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.808871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.16.190.195:42151)
0927 21:05:47.808935 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.808936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.808946 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:47.808987 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.808988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.809005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.809075 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:47.809204 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.809706 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.810640 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.810840 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.810948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.811087 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:47.811315 (+ 228us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:47.811389 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.811447 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.811473 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:05:47.813606 1958 raft_consensus.cc:1261] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 1 FOLLOWER]: Refusing update from remote peer 78b52b29dd8c409f9fc19ac7297b0f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:47.813627 1823 raft_consensus.cc:1261] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7 [term 1 FOLLOWER]: Refusing update from remote peer 78b52b29dd8c409f9fc19ac7297b0f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:47.814052 2166 consensus_queue.cc:1035] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "673d0be286424eafa837a852f9cb5ec7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:47.814210 2164 consensus_queue.cc:1035] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd82b6531a1b478e96c266d79acc5b6a" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:47.816608 2169 mvcc.cc:205] Tried to move back new op lower bound from 7334064733441380352 to 7334064733296349184. Current Snapshot: MvccSnapshot[applied={T|T < 7334064733441380352}]
I20260927 21:05:47.817061 2170 mvcc.cc:205] Tried to move back new op lower bound from 7334064733441380352 to 7334064733296349184. Current Snapshot: MvccSnapshot[applied={T|T < 7334064733441380352}]
I20260927 21:05:47.817245 2171 mvcc.cc:205] Tried to move back new op lower bound from 7334064733441380352 to 7334064733296349184. Current Snapshot: MvccSnapshot[applied={T|T < 7334064733441380352}]
I20260927 21:05:47.819129 1677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:47.819247 1677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:47.820104 1677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ff04e5da89a499daa902a810b05acc3: Sending DeleteTablet for 3 replicas of tablet fe73816f00c842f2a0e7f5faba517c07
I20260927 21:05:47.820464 1938 tablet_service.cc:1514] Processing DeleteTablet for tablet fe73816f00c842f2a0e7f5faba517c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:47 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:50450
I20260927 21:05:47.820518 1803 tablet_service.cc:1514] Processing DeleteTablet for tablet fe73816f00c842f2a0e7f5faba517c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:47 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:55080
I20260927 21:05:47.820510 2072 tablet_service.cc:1514] Processing DeleteTablet for tablet fe73816f00c842f2a0e7f5faba517c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:47 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:59832
I20260927 21:05:47.820750 2181 tablet_replica.cc:330] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: stopping tablet replica
I20260927 21:05:47.820884 2181 raft_consensus.cc:2229] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:47.821023 2181 raft_consensus.cc:2258] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:47.821130 2180 tablet_replica.cc:330] T fe73816f00c842f2a0e7f5faba517c07 P 673d0be286424eafa837a852f9cb5ec7: stopping tablet replica
I20260927 21:05:47.821202 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1739
I20260927 21:05:47.821894 2181 ts_tablet_manager.cc:1905] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:47.824272 2181 ts_tablet_manager.cc:1918] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:47.824361 2181 log.cc:1196] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/fe73816f00c842f2a0e7f5faba517c07
I20260927 21:05:47.824658 2181 ts_tablet_manager.cc:1939] T fe73816f00c842f2a0e7f5faba517c07 P 78b52b29dd8c409f9fc19ac7297b0f61: Deleting consensus metadata
I20260927 21:05:47.825145 1661 catalog_manager.cc:4841] TS 78b52b29dd8c409f9fc19ac7297b0f61 (127.16.190.195:42151): tablet fe73816f00c842f2a0e7f5faba517c07 (table test-table [id=774ffe0649824246bb0ab9b1bb6ad3f7]) successfully deleted
W20260927 21:05:47.826948 1661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:37793 (error 108)
I20260927 21:05:47.827487 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.827342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:37793 (local address 127.0.0.1:55094)
0927 21:05:47.827404 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.827451 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:37793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 21:05:47.827682 1661 catalog_manager.cc:4584] TS 673d0be286424eafa837a852f9cb5ec7 (127.16.190.193:37793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe73816f00c842f2a0e7f5faba517c07: Network error: Client connection negotiation failed: client connection to 127.16.190.193:37793: connect: Connection refused (error 111)
I20260927 21:05:47.827999 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1874
I20260927 21:05:47.828011 2179 tablet_replica.cc:330] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a: stopping tablet replica
I20260927 21:05:47.828157 2179 raft_consensus.cc:2229] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:47.828279 2179 raft_consensus.cc:2258] T fe73816f00c842f2a0e7f5faba517c07 P fd82b6531a1b478e96c266d79acc5b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:47.833456 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.833346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44151 (local address 127.0.0.1:50460)
0927 21:05:47.833384 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:47.833416 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.194:44151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 21:05:47.833549 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2009
I20260927 21:05:47.839259 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 1644
2026-09-27T21:05:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T21:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:47Z Disabled control of system clock
I20260927 21:05:47.867324 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:35199
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:47.948729 2186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:47.948999 2186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:47.949021 2186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:47.950279 2186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:47.950330 2186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:47.950341 2186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:47.950354 2186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:47.950366 2186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:47.951874 2186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:35199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:47.952150 2186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:47.952466 2186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:47.957495 2192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:47.957491 2194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.957620 2186 server_base.cc:1034] running on GCE node
W20260927 21:05:47.957742 2191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:47.958084 2186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:47.958338 2186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:47.959525 2186 hybrid_clock.cc:648] HybridClock initialized: now 1790543147959510 us; error 63 us; skew 500 ppm
I20260927 21:05:47.960868 2186 webserver.cc:460] Webserver started at http://127.16.190.254:40827/ using document root <none> and password file <none>
I20260927 21:05:47.961167 2186 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:47.961244 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:47.961398 2186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:47.962575 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "6df246b8469741468efb3a2a442f110b"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "88ff9cdd6a961342022ea495a86217df"
server_key_iv: "ead24f2d04c27f7fdfea3f788b520b6c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.962960 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "6df246b8469741468efb3a2a442f110b"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "88ff9cdd6a961342022ea495a86217df"
server_key_iv: "ead24f2d04c27f7fdfea3f788b520b6c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.964319 2186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:47.965114 2200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:47.965440 2186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:47.965559 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "6df246b8469741468efb3a2a442f110b"
format_stamp: "Formatted at 2026-09-27 21:05:47 on dist-test-slave-6c7s"
server_key: "88ff9cdd6a961342022ea495a86217df"
server_key_iv: "ead24f2d04c27f7fdfea3f788b520b6c"
server_key_version: "encryptionkey@0"
I20260927 21:05:47.965641 2186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:47.986146 2186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:47.986507 2186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:47.986670 2186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:47.990705 2186 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:35199
I20260927 21:05:47.990743 2252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:35199 every 8 connection(s)
I20260927 21:05:47.991132 2186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:47.992143 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:47.992220 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2186
I20260927 21:05:47.992352 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:47.992614 17147 external_mini_cluster.cc:1428] Setting key a2d5b6f740bc396828048ebf82483df5
I20260927 21:05:47.995877 2253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: Bootstrap starting.
I20260927 21:05:47.996562 2253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:47.996892 2253 log.cc:824] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:47.997761 2253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: No bootstrap required, opened a new log
I20260927 21:05:47.998874 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:47.993217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.16.190.254:35199)
0927 21:05:47.993441 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:47.993452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:47.993488 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:05:47.993983 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:47.993993 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:47.994083 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:47.994242 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:47.994422 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.995812 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.996730 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:47.997232 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:47.997330 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:47.998107 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:47.998112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:47.998112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:47.998319 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:47.998363 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:47.998379 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:47.998458 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:47.998563 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:05:47.998643 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260927 21:05:48.000138 2253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } }
I20260927 21:05:48.000285 2253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:48.000308 2253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df246b8469741468efb3a2a442f110b, State: Initialized, Role: FOLLOWER
I20260927 21:05:48.000447 2253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } }
I20260927 21:05:48.000533 2253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:48.000558 2253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:48.000614 2253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:48.001313 2253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } }
I20260927 21:05:48.001410 2253 leader_election.cc:302] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df246b8469741468efb3a2a442f110b; no voters:
I20260927 21:05:48.001639 2253 leader_election.cc:288] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:48.001740 2259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:48.001888 2259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [term 1 LEADER]: Becoming Leader. State: Replica: 6df246b8469741468efb3a2a442f110b, State: Running, Role: LEADER
I20260927 21:05:48.001932 2253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:48.001978 2259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } }
I20260927 21:05:48.002400 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df246b8469741468efb3a2a442f110b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } } }
I20260927 21:05:48.002425 2261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df246b8469741468efb3a2a442f110b. Latest consensus state: current_term: 1 leader_uuid: "6df246b8469741468efb3a2a442f110b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df246b8469741468efb3a2a442f110b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 35199 } } }
I20260927 21:05:48.002595 2261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:48.002825 2268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:48.002972 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:48.003328 2268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:48.005003 2268 catalog_manager.cc:1347] Generated new cluster ID: 4c6c614053d64ea9ac3fa9191512e31e
I20260927 21:05:48.005070 2268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:48.009927 2268 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:48.010479 2268 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:48.015483 2268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: Generated new TSK 0
I20260927 21:05:48.015689 2268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:48.017607 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--builtin_ntp_servers=127.16.190.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:48.119668 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:48.119894 2278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:48.119917 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:48.121253 2278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:48.121297 2278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:48.121325 2278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:48.122713 2278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:48.122965 2278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:48.123301 2278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:48.129328 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:48.129312 2284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:48.129326 2283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:48.129942 2278 server_base.cc:1034] running on GCE node
I20260927 21:05:48.130153 2278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:48.130400 2278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:48.131579 2278 hybrid_clock.cc:648] HybridClock initialized: now 1790543148131555 us; error 45 us; skew 500 ppm
I20260927 21:05:48.132992 2278 webserver.cc:460] Webserver started at http://127.16.190.193:39569/ using document root <none> and password file <none>
I20260927 21:05:48.133248 2278 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:48.133320 2278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:48.133461 2278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:48.134555 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "e5e4c85bc3ae468cbb109a820a74d2b7"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "08ed2fd1b578b910a207de88cc853a24"
server_key_iv: "140cbf76db291ef70c4f6467b8bda5f8"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.134931 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e4c85bc3ae468cbb109a820a74d2b7"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "08ed2fd1b578b910a207de88cc853a24"
server_key_iv: "140cbf76db291ef70c4f6467b8bda5f8"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.136309 2278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:48.137140 2292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.137336 2278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:48.137439 2278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "e5e4c85bc3ae468cbb109a820a74d2b7"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "08ed2fd1b578b910a207de88cc853a24"
server_key_iv: "140cbf76db291ef70c4f6467b8bda5f8"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.137526 2278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:48.154718 2278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:48.155076 2278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:48.155237 2278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:48.155485 2278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:48.155823 2278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:48.155879 2278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.155948 2278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:48.155979 2278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.162099 2278 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:40065
I20260927 21:05:48.162179 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:40065 every 8 connection(s)
I20260927 21:05:48.162570 2278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:48.163708 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2278
I20260927 21:05:48.163808 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:48.163997 17147 external_mini_cluster.cc:1428] Setting key 22c705fb9f52933a882df4a2e6af100e
I20260927 21:05:48.165072 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--builtin_ntp_servers=127.16.190.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:05:48.168776 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.163384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:45015 (local address 127.16.190.254:35199)
0927 21:05:48.163458 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.163460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.163711 (+ 251us) server_negotiation.cc:407] Connection header received
0927 21:05:48.164344 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.164345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.164376 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.164441 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:48.165272 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.165785 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.167258 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.167449 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.167509 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.168463 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:48.168465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:48.168465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:48.168469 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:48.168483 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:48.168487 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:48.168532 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.168655 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.168689 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:05:48.169095 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.163344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:35199 (local address 127.16.190.193:45015)
0927 21:05:48.163574 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.163610 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.164228 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.164495 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.164499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.164507 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:48.165145 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.165151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.165879 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.165880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.167166 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.167169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.167540 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.167541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.167717 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.168299 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:48.168300 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:48.168340 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:48.168371 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:48.168550 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:48.168575 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:48.168578 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.168655 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.168777 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
I20260927 21:05:48.169718 2408 heartbeater.cc:347] Connected to a master server at 127.16.190.254:35199
I20260927 21:05:48.169817 2408 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:48.170038 2408 heartbeater.cc:511] Master 127.16.190.254:35199 requested a full tablet report, sending...
I20260927 21:05:48.170573 2217 ts_manager.cc:194] Registered new tserver with Master: e5e4c85bc3ae468cbb109a820a74d2b7 (127.16.190.193:40065)
I20260927 21:05:48.171211 2217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.193:45015
W20260927 21:05:48.250674 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:48.250941 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:48.250976 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:48.252249 2411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:48.252329 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:48.252374 2411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:48.253813 2411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:48.254103 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:48.254422 2411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:48.259593 2419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:48.259593 2417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:48.259846 2411 server_base.cc:1034] running on GCE node
W20260927 21:05:48.260038 2416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:48.260349 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:48.260576 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:48.261724 2411 hybrid_clock.cc:648] HybridClock initialized: now 1790543148261698 us; error 50 us; skew 500 ppm
I20260927 21:05:48.263051 2411 webserver.cc:460] Webserver started at http://127.16.190.194:39745/ using document root <none> and password file <none>
I20260927 21:05:48.263303 2411 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:48.263356 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:48.263535 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:48.264650 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "ab7cfb6fab59465d85df0d005f2f9801"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "8843edf2d31b16eadc5b981b58698c48"
server_key_iv: "64dd609284f2d3e4378ea994d354cd26"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.265058 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "ab7cfb6fab59465d85df0d005f2f9801"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "8843edf2d31b16eadc5b981b58698c48"
server_key_iv: "64dd609284f2d3e4378ea994d354cd26"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.266373 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:48.267076 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.267247 2411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:05:48.267345 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "ab7cfb6fab59465d85df0d005f2f9801"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "8843edf2d31b16eadc5b981b58698c48"
server_key_iv: "64dd609284f2d3e4378ea994d354cd26"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.267431 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:48.289116 2411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:48.289460 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:48.289618 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:48.289883 2411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:48.290231 2411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:48.290285 2411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.290326 2411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:48.290354 2411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.296350 2411 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:34379
I20260927 21:05:48.296411 2539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:34379 every 8 connection(s)
I20260927 21:05:48.296787 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:48.300974 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2411
I20260927 21:05:48.301096 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:48.301376 17147 external_mini_cluster.cc:1428] Setting key a269c7d8f9313cc0f671b2317243a662
I20260927 21:05:48.302531 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.297604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:49243 (local address 127.16.190.254:35199)
0927 21:05:48.297706 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.297708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.297944 (+ 236us) server_negotiation.cc:407] Connection header received
0927 21:05:48.298501 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.298503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.298515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.298568 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:48.299288 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.299791 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.300874 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.301061 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.301159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.302214 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:48.302215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:48.302216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:48.302220 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:48.302241 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:48.302245 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:48.302291 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.302440 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.302471 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:05:48.302525 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--builtin_ntp_servers=127.16.190.212:38003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:05:48.302806 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.297563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:35199 (local address 127.16.190.194:49243)
0927 21:05:48.297816 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.297853 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.298386 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.298649 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.298653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.298661 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:48.299178 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.299183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.299887 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.299889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.300737 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.300739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.301198 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.301199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.301403 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.302044 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:48.302045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:48.302085 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:48.302115 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:48.302308 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:48.302312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:48.302316 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.302415 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.302533 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260927 21:05:48.304167 2540 heartbeater.cc:347] Connected to a master server at 127.16.190.254:35199
I20260927 21:05:48.304262 2540 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:48.304461 2540 heartbeater.cc:511] Master 127.16.190.254:35199 requested a full tablet report, sending...
I20260927 21:05:48.304878 2217 ts_manager.cc:194] Registered new tserver with Master: ab7cfb6fab59465d85df0d005f2f9801 (127.16.190.194:34379)
I20260927 21:05:48.305269 2217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.194:49243
W20260927 21:05:48.385761 2543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:48.386036 2543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:48.386076 2543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:48.387283 2543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:48.387352 2543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:48.387400 2543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:48.388794 2543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:48.389122 2543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:48.389433 2543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:48.394515 2551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:48.394515 2548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:48.394520 2549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:48.394785 2543 server_base.cc:1034] running on GCE node
I20260927 21:05:48.395018 2543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:48.395274 2543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:48.396448 2543 hybrid_clock.cc:648] HybridClock initialized: now 1790543148396428 us; error 38 us; skew 500 ppm
I20260927 21:05:48.397889 2543 webserver.cc:460] Webserver started at http://127.16.190.195:45383/ using document root <none> and password file <none>
I20260927 21:05:48.398135 2543 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:48.398205 2543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:48.398365 2543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:48.399928 2543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "4c25c0109ae84cd7977dba960bc40335"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "e1cfc8e6046b09f2370b8b08523db11e"
server_key_iv: "0bf3f888625e705c936351c9f13e0919"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.400305 2543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "4c25c0109ae84cd7977dba960bc40335"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "e1cfc8e6046b09f2370b8b08523db11e"
server_key_iv: "0bf3f888625e705c936351c9f13e0919"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.401752 2543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 21:05:48.402493 2557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.402711 2543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:48.402791 2543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "4c25c0109ae84cd7977dba960bc40335"
format_stamp: "Formatted at 2026-09-27 21:05:48 on dist-test-slave-6c7s"
server_key: "e1cfc8e6046b09f2370b8b08523db11e"
server_key_iv: "0bf3f888625e705c936351c9f13e0919"
server_key_version: "encryptionkey@0"
I20260927 21:05:48.402846 2543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:48.412895 2543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:48.413228 2543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:48.413410 2543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:48.413652 2543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:48.413995 2543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:48.414027 2543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.414075 2543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:48.414103 2543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:48.420030 2543 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:39573
I20260927 21:05:48.420104 2671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:39573 every 8 connection(s)
I20260927 21:05:48.420451 2543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:48.425652 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.421283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:52419 (local address 127.16.190.254:35199)
0927 21:05:48.421354 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.421356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.421628 (+ 272us) server_negotiation.cc:407] Connection header received
0927 21:05:48.422120 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.422121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.422131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.422180 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:48.422823 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.423306 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.424255 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.424463 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.424517 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.425405 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:48.425406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:48.425407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:48.425411 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:48.425424 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:48.425429 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:48.425474 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.425563 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.425586 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:05:48.425865 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.421292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:35199 (local address 127.16.190.195:52419)
0927 21:05:48.421506 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.421540 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.422037 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.422242 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.422246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.422254 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:48.422755 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.422759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.423391 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.423392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.424182 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.424184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.424541 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.424542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.424654 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.425244 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:48.425244 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:48.425274 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:48.425297 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:48.425487 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:48.425491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:48.425493 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.425551 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.425635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260927 21:05:48.426359 2672 heartbeater.cc:347] Connected to a master server at 127.16.190.254:35199
I20260927 21:05:48.426460 2672 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:48.426611 2672 heartbeater.cc:511] Master 127.16.190.254:35199 requested a full tablet report, sending...
I20260927 21:05:48.426995 2217 ts_manager.cc:194] Registered new tserver with Master: 4c25c0109ae84cd7977dba960bc40335 (127.16.190.195:39573)
I20260927 21:05:48.427325 2217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.195:52419
I20260927 21:05:48.427821 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2543
I20260927 21:05:48.427943 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:05:48.428136 17147 external_mini_cluster.cc:1428] Setting key cbe5e2cc2e4123d81d21a12278179b34
I20260927 21:05:48.429126 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:48.433133 2257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.430480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.16.190.254:35199)
0927 21:05:48.430559 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.430561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.430602 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:05:48.430642 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.430643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.430652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.430703 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:48.430817 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.431269 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.431995 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.432169 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.432219 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.432860 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:48.432861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:48.432861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:48.432866 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:48.432880 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:48.432885 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:48.432938 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.433002 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.433021 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 21:05:48.434986 2217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:48.436519 2217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:48.442952 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.438641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34379 (local address 127.0.0.1:50820)
0927 21:05:48.438814 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.438838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.438917 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.439315 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.439318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.439320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:48.439394 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.439395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440788 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.440789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.441789 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.441791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.442627 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.442709 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.442723 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.442799 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.442866 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260927 21:05:48.442967 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.438757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.16.190.195:39573)
0927 21:05:48.439190 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.439195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.439205 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:48.439242 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.439244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.439260 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.439360 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:48.439568 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440414 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.441976 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.442511 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442551 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.442654 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.442855 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.442892 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":98,"threads_started":1}
I20260927 21:05:48.442950 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.438955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.16.190.194:34379)
0927 21:05:48.439148 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.439153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.439164 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:48.439202 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.439205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.439224 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.439312 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:48.439513 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440698 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.441887 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.442537 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442613 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.442696 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.442815 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.442859 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260927 21:05:48.443418 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.438862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:40065 (local address 127.0.0.1:38414)
0927 21:05:48.439064 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.439075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.439121 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.439718 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.439719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.439720 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:48.439755 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.439756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440981 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.440982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.442064 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.442065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.443197 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.443198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.443257 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.443262 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.443316 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.443354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":63,"threads_started":1}
I20260927 21:05:48.443507 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.438641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:39573 (local address 127.0.0.1:50216)
0927 21:05:48.438794 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.438837 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.438901 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.439428 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.439430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.439432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:48.439478 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.439480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440504 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.440505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.441878 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.441881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.442711 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.442766 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.442769 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.443367 (+ 598us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.443413 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":66,"thread_start_us":71,"threads_started":1}
I20260927 21:05:48.443776 2474 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2fe1993984af79136c6e446307508 (DEFAULT_TABLE table=test-table [id=09c410e377c9409aadb142e48441329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:48.444109 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.439054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.16.190.193:40065)
0927 21:05:48.439476 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.439513 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.439522 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:05:48.439562 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.439564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.439579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.439650 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:48.439840 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.440898 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442145 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.442826 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.442899 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.442975 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.443986 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.444033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":78,"threads_started":1}
I20260927 21:05:48.444128 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2fe1993984af79136c6e446307508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:48.444321 2606 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2fe1993984af79136c6e446307508 (DEFAULT_TABLE table=test-table [id=09c410e377c9409aadb142e48441329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:48.444653 2606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2fe1993984af79136c6e446307508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:48.444773 2341 tablet_service.cc:1467] Processing CreateTablet for tablet 41f2fe1993984af79136c6e446307508 (DEFAULT_TABLE table=test-table [id=09c410e377c9409aadb142e48441329b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:48.445168 2341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f2fe1993984af79136c6e446307508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:48.446781 2686 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Bootstrap starting.
I20260927 21:05:48.447352 2686 tablet_bootstrap.cc:654] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:48.447566 2686 log.cc:824] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:48.448019 2688 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Bootstrap starting.
I20260927 21:05:48.448247 2686 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: No bootstrap required, opened a new log
I20260927 21:05:48.448308 2686 ts_tablet_manager.cc:1397] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:48.448712 2689 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: Bootstrap starting.
I20260927 21:05:48.448949 2688 tablet_bootstrap.cc:654] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:48.449318 2689 tablet_bootstrap.cc:654] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:48.449376 2688 log.cc:824] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:48.449657 2689 log.cc:824] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:48.450182 2688 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: No bootstrap required, opened a new log
I20260927 21:05:48.450253 2688 ts_tablet_manager.cc:1397] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:48.451210 2686 raft_consensus.cc:348] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.451404 2686 raft_consensus.cc:374] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:48.451442 2686 raft_consensus.cc:729] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7cfb6fab59465d85df0d005f2f9801, State: Initialized, Role: FOLLOWER
I20260927 21:05:48.451555 2686 consensus_queue.cc:260] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.451804 2686 ts_tablet_manager.cc:1428] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 21:05:48.451900 2540 heartbeater.cc:503] Master 127.16.190.254:35199 was elected leader, sending a full tablet report...
I20260927 21:05:48.453050 2688 raft_consensus.cc:348] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.453233 2688 raft_consensus.cc:374] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:48.453284 2688 raft_consensus.cc:729] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c25c0109ae84cd7977dba960bc40335, State: Initialized, Role: FOLLOWER
I20260927 21:05:48.453403 2688 consensus_queue.cc:260] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.453455 2689 tablet_bootstrap.cc:492] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: No bootstrap required, opened a new log
I20260927 21:05:48.453522 2689 ts_tablet_manager.cc:1397] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 21:05:48.453672 2688 ts_tablet_manager.cc:1428] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:48.453691 2672 heartbeater.cc:503] Master 127.16.190.254:35199 was elected leader, sending a full tablet report...
I20260927 21:05:48.455394 2689 raft_consensus.cc:348] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.455555 2689 raft_consensus.cc:374] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:48.455603 2689 raft_consensus.cc:729] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e4c85bc3ae468cbb109a820a74d2b7, State: Initialized, Role: FOLLOWER
I20260927 21:05:48.455739 2689 consensus_queue.cc:260] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.456056 2689 ts_tablet_manager.cc:1428] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 21:05:48.456095 2408 heartbeater.cc:503] Master 127.16.190.254:35199 was elected leader, sending a full tablet report...
W20260927 21:05:48.547427 2541 tablet.cc:2369] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:05:48.583154 2693 raft_consensus.cc:482] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:48.583359 2693 raft_consensus.cc:504] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.583798 2693 leader_election.cc:288] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e4c85bc3ae468cbb109a820a74d2b7 (127.16.190.193:40065), ab7cfb6fab59465d85df0d005f2f9801 (127.16.190.194:34379)
I20260927 21:05:48.587082 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.583988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:40065 (local address 127.16.190.195:42217)
0927 21:05:48.584061 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.584076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.584128 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.584320 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.584321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.584323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:48.584400 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.584402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.585379 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.585381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.586171 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.586173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.586807 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.586888 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.586890 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.586938 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.586987 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":13}
I20260927 21:05:48.587153 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.584057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:42217 (local address 127.16.190.193:40065)
0927 21:05:48.584106 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.584109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.584121 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:48.584198 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.584199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.584230 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.584295 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:48.584502 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.585272 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586266 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.586711 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586786 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.586795 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.587035 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.587077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:05:48.587334 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.584002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34379 (local address 127.16.190.195:43501)
0927 21:05:48.584191 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.584201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.584272 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.584425 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:48.584426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:48.584427 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:48.584521 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.584523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.585412 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.585413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.586171 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:48.586173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.586789 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:48.586867 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.586873 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 21:05:48.586957 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:05:48.586990 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260927 21:05:48.587352 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.584093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:43501 (local address 127.16.190.194:34379)
0927 21:05:48.584141 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.584143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.584304 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:05:48.584336 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.584337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.584347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.584407 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:48.584629 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.585316 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586270 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.586652 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.586729 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.586737 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.587001 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.587084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:05:48.587548 2361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2fe1993984af79136c6e446307508" candidate_uuid: "4c25c0109ae84cd7977dba960bc40335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" is_pre_election: true
I20260927 21:05:48.587707 2361 raft_consensus.cc:2454] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c25c0109ae84cd7977dba960bc40335 in term 0.
I20260927 21:05:48.587970 2561 leader_election.cc:302] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c25c0109ae84cd7977dba960bc40335, e5e4c85bc3ae468cbb109a820a74d2b7; no voters:
I20260927 21:05:48.587978 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2fe1993984af79136c6e446307508" candidate_uuid: "4c25c0109ae84cd7977dba960bc40335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7cfb6fab59465d85df0d005f2f9801" is_pre_election: true
I20260927 21:05:48.588131 2494 raft_consensus.cc:2454] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c25c0109ae84cd7977dba960bc40335 in term 0.
I20260927 21:05:48.588145 2693 raft_consensus.cc:2781] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:48.588238 2693 raft_consensus.cc:482] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:48.588274 2693 raft_consensus.cc:3037] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:48.589099 2693 raft_consensus.cc:504] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.589236 2693 leader_election.cc:288] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [CANDIDATE]: Term 1 election: Requested vote from peers e5e4c85bc3ae468cbb109a820a74d2b7 (127.16.190.193:40065), ab7cfb6fab59465d85df0d005f2f9801 (127.16.190.194:34379)
I20260927 21:05:48.589401 2494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2fe1993984af79136c6e446307508" candidate_uuid: "4c25c0109ae84cd7977dba960bc40335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7cfb6fab59465d85df0d005f2f9801"
I20260927 21:05:48.589433 2361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f2fe1993984af79136c6e446307508" candidate_uuid: "4c25c0109ae84cd7977dba960bc40335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7"
I20260927 21:05:48.589494 2494 raft_consensus.cc:3037] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:48.589534 2361 raft_consensus.cc:3037] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:48.590235 2494 raft_consensus.cc:2454] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c25c0109ae84cd7977dba960bc40335 in term 1.
I20260927 21:05:48.590378 2361 raft_consensus.cc:2454] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c25c0109ae84cd7977dba960bc40335 in term 1.
I20260927 21:05:48.590437 2559 leader_election.cc:302] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c25c0109ae84cd7977dba960bc40335, ab7cfb6fab59465d85df0d005f2f9801; no voters:
I20260927 21:05:48.590528 2693 raft_consensus.cc:2781] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:48.590670 2693 raft_consensus.cc:686] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 1 LEADER]: Becoming Leader. State: Replica: 4c25c0109ae84cd7977dba960bc40335, State: Running, Role: LEADER
I20260927 21:05:48.590777 2693 consensus_queue.cc:237] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } }
I20260927 21:05:48.591739 2217 catalog_manager.cc:5495] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c25c0109ae84cd7977dba960bc40335 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "4c25c0109ae84cd7977dba960bc40335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c25c0109ae84cd7977dba960bc40335" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 39573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 } health_report { overall_health: UNKNOWN } } }
I20260927 21:05:48.625648 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:48.623146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.16.190.195:39573)
0927 21:05:48.623223 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:48.623225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:48.623235 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:48.623287 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:48.623288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:48.623306 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:48.623372 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:48.623528 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.624014 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.624805 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:48.625035 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:48.625099 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:48.625222 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:48.625330 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:48.625389 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:48.625442 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:05:48.625482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:05:48.627616 2494 raft_consensus.cc:1261] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 1 FOLLOWER]: Refusing update from remote peer 4c25c0109ae84cd7977dba960bc40335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:48.627629 2361 raft_consensus.cc:1261] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 1 FOLLOWER]: Refusing update from remote peer 4c25c0109ae84cd7977dba960bc40335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:48.628041 2696 consensus_queue.cc:1035] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e4c85bc3ae468cbb109a820a74d2b7" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:48.628194 2693 consensus_queue.cc:1035] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7cfb6fab59465d85df0d005f2f9801" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:48.631708 2700 mvcc.cc:205] Tried to move back new op lower bound from 7334064736775598080 to 7334064736628613120. Current Snapshot: MvccSnapshot[applied={T|T < 7334064736775598080}]
I20260927 21:05:48.631963 2701 mvcc.cc:205] Tried to move back new op lower bound from 7334064736775598080 to 7334064736628613120. Current Snapshot: MvccSnapshot[applied={T|T < 7334064736775598080 or (T in {7334064736775598080})}]
I20260927 21:05:48.633436 2217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:48.633536 2217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:48.634347 2217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6df246b8469741468efb3a2a442f110b: Sending DeleteTablet for 3 replicas of tablet 41f2fe1993984af79136c6e446307508
I20260927 21:05:48.634658 2606 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f2fe1993984af79136c6e446307508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:48 UTC) from {username='slave'} at 127.0.0.1:50216
I20260927 21:05:48.634709 2341 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f2fe1993984af79136c6e446307508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:48 UTC) from {username='slave'} at 127.0.0.1:38414
I20260927 21:05:48.634693 2474 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f2fe1993984af79136c6e446307508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:48 UTC) from {username='slave'} at 127.0.0.1:50820
I20260927 21:05:48.635018 2709 tablet_replica.cc:330] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: stopping tablet replica
I20260927 21:05:48.635027 2710 tablet_replica.cc:330] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: stopping tablet replica
I20260927 21:05:48.635114 2709 raft_consensus.cc:2229] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:48.635116 2710 raft_consensus.cc:2229] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:48.635215 2710 raft_consensus.cc:2258] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:48.635234 2709 raft_consensus.cc:2258] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:48.635264 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2278
I20260927 21:05:48.635435 2711 tablet_replica.cc:330] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7: stopping tablet replica
I20260927 21:05:48.635520 2711 raft_consensus.cc:2229] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:48.635607 2711 raft_consensus.cc:2258] T 41f2fe1993984af79136c6e446307508 P e5e4c85bc3ae468cbb109a820a74d2b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:48.636299 2710 ts_tablet_manager.cc:1905] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:48.636550 2709 ts_tablet_manager.cc:1905] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:48.638087 2710 ts_tablet_manager.cc:1918] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:48.638164 2710 log.cc:1196] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/41f2fe1993984af79136c6e446307508
I20260927 21:05:48.638429 2710 ts_tablet_manager.cc:1939] T 41f2fe1993984af79136c6e446307508 P ab7cfb6fab59465d85df0d005f2f9801: Deleting consensus metadata
I20260927 21:05:48.638604 2709 ts_tablet_manager.cc:1918] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:48.638669 2709 log.cc:1196] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/41f2fe1993984af79136c6e446307508
I20260927 21:05:48.639079 2202 catalog_manager.cc:4841] TS ab7cfb6fab59465d85df0d005f2f9801 (127.16.190.194:34379): tablet 41f2fe1993984af79136c6e446307508 (table test-table [id=09c410e377c9409aadb142e48441329b]) successfully deleted
I20260927 21:05:48.638928 2709 ts_tablet_manager.cc:1939] T 41f2fe1993984af79136c6e446307508 P 4c25c0109ae84cd7977dba960bc40335: Deleting consensus metadata
I20260927 21:05:48.639670 2204 catalog_manager.cc:4841] TS 4c25c0109ae84cd7977dba960bc40335 (127.16.190.195:39573): tablet 41f2fe1993984af79136c6e446307508 (table test-table [id=09c410e377c9409aadb142e48441329b]) successfully deleted
W20260927 21:05:48.641929 2204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:40065 (error 108)
W20260927 21:05:48.642733 2683 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:05:48.642326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:40065 (local address 127.0.0.1:38416)
0927 21:05:48.642376 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:48.642390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:48.642435 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:48.642656 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:40065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 21:05:48.642928 2204 catalog_manager.cc:4584] TS e5e4c85bc3ae468cbb109a820a74d2b7 (127.16.190.193:40065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41f2fe1993984af79136c6e446307508: Network error: Client connection negotiation failed: client connection to 127.16.190.193:40065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:05:48.642962 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2411
I20260927 21:05:48.648334 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2543
I20260927 21:05:48.653250 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2186
2026-09-27T21:05:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (814 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:48 dist-test-slave-6c7s krb5kdc[2715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:48 dist-test-slave-6c7s krb5kdc[2715](info): set up 2 sockets
Sep 27 21:05:48 dist-test-slave-6c7s krb5kdc[2715](info): commencing operation
krb5kdc: starting...
W20260927 21:05:49.725448 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:49 dist-test-slave-6c7s krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:05:49.853446 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45549
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:49.936550 2729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:49.936794 2729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:49.936851 2729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:49.938225 2729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:49.938277 2729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:49.938292 2729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:49.938305 2729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:49.938318 2729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:49.939961 2729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:45549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:49.940265 2729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:49.940518 2729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:49.945992 2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:49.946123 2729 server_base.cc:1034] running on GCE node
W20260927 21:05:49.946179 2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:49.945974 2737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:49.946501 2729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:49.946820 2729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:49.948002 2729 hybrid_clock.cc:648] HybridClock initialized: now 1790543149947985 us; error 44 us; skew 500 ppm
Sep 27 21:05:49 dist-test-slave-6c7s krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543149, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:49.950985 2729 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:05:49.951637 2729 webserver.cc:460] Webserver started at http://127.16.190.254:46005/ using document root <none> and password file <none>
I20260927 21:05:49.951911 2729 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:49.951959 2729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:49.952090 2729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:49.953392 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "85f16da5fcc540f2955ca109052e94ed"
format_stamp: "Formatted at 2026-09-27 21:05:49 on dist-test-slave-6c7s"
server_key: "fa6804560acec8755de1090e343ad8ad"
server_key_iv: "c0d0d68cb503d7c7948889fc66a1f8ce"
server_key_version: "encryptionkey@0"
I20260927 21:05:49.953789 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "85f16da5fcc540f2955ca109052e94ed"
format_stamp: "Formatted at 2026-09-27 21:05:49 on dist-test-slave-6c7s"
server_key: "fa6804560acec8755de1090e343ad8ad"
server_key_iv: "c0d0d68cb503d7c7948889fc66a1f8ce"
server_key_version: "encryptionkey@0"
I20260927 21:05:49.955307 2729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:05:49.956171 2744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:49.956391 2729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:49.956497 2729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "85f16da5fcc540f2955ca109052e94ed"
format_stamp: "Formatted at 2026-09-27 21:05:49 on dist-test-slave-6c7s"
server_key: "fa6804560acec8755de1090e343ad8ad"
server_key_iv: "c0d0d68cb503d7c7948889fc66a1f8ce"
server_key_version: "encryptionkey@0"
I20260927 21:05:49.956593 2729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:49.977363 2729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:49.980082 2729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:49.980396 2729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:49.985179 2729 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:45549
I20260927 21:05:49.985227 2796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:45549 every 8 connection(s)
I20260927 21:05:49.985632 2729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:49.986538 2797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:49.988860 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2729
I20260927 21:05:49.988919 2797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: Bootstrap starting.
I20260927 21:05:49.989010 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:49.989210 17147 external_mini_cluster.cc:1428] Setting key d0422e7c20e4e25f77cb23241e10f287
I20260927 21:05:49.989641 2797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:49.990020 2797 log.cc:824] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:49.990834 2797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: No bootstrap required, opened a new log
I20260927 21:05:49.993196 2797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } }
I20260927 21:05:49.993348 2797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:49.993373 2797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f16da5fcc540f2955ca109052e94ed, State: Initialized, Role: FOLLOWER
I20260927 21:05:49.993503 2797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } }
I20260927 21:05:49.993575 2797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:49.993638 2797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:49.993686 2797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:49.994390 2797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } }
I20260927 21:05:49.994493 2797 leader_election.cc:302] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f16da5fcc540f2955ca109052e94ed; no voters:
I20260927 21:05:49.994657 2797 leader_election.cc:288] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:49.994716 2802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:49.994869 2802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [term 1 LEADER]: Becoming Leader. State: Replica: 85f16da5fcc540f2955ca109052e94ed, State: Running, Role: LEADER
I20260927 21:05:49.994930 2797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:49.994961 2802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } }
I20260927 21:05:49.995285 2802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f16da5fcc540f2955ca109052e94ed. Latest consensus state: current_term: 1 leader_uuid: "85f16da5fcc540f2955ca109052e94ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } } }
I20260927 21:05:49.995363 2802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:49.995294 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f16da5fcc540f2955ca109052e94ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f16da5fcc540f2955ca109052e94ed" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 45549 } } }
I20260927 21:05:49.995414 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed [sys.catalog]: This master's current role is: LEADER
W20260927 21:05:49.996899 2818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:05:49.996958 2818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:05:49.997012 2808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:49.997205 2808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 21:05:49 dist-test-slave-6c7s krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:49.999091 2808 catalog_manager.cc:1347] Generated new cluster ID: a37700ed495f4794bd654ad5268b8e0c
I20260927 21:05:49.999150 2808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:50.002511 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:49.989795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.16.190.254:45549)
0927 21:05:49.989976 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:49.989986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:49.990020 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:05:49.990633 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:49.990643 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:49.990707 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:49.990866 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:49.991529 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:49.993057 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:49.994031 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:49.994697 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:49.994817 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:49.998101 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:49.998104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:49.998105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:49.998117 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.000210 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.000712 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.000716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.000716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.000862 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.001093 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.001094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.001094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.001431 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.002001 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.002089 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.002245 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260927 21:05:50.008086 2808 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:50.008857 2808 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:50.019510 2808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: Generated new TSK 0
I20260927 21:05:50.019726 2808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:05:50.056483 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:50.139806 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:50.140074 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:50.140115 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:50.141358 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:50.141431 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:50.141479 2823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:50.143047 2823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:50.143326 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:50.143554 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:50.148672 2829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.148691 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.148681 2828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.148923 2823 server_base.cc:1034] running on GCE node
I20260927 21:05:50.149140 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:50.149401 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:50.150589 2823 hybrid_clock.cc:648] HybridClock initialized: now 1790543150150575 us; error 29 us; skew 500 ppm
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:50.153527 2823 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:05:50.153991 2823 webserver.cc:460] Webserver started at http://127.16.190.193:36287/ using document root <none> and password file <none>
I20260927 21:05:50.154219 2823 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:50.154265 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:50.154400 2823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:50.155493 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "a66f823c20b1493cbaa4747eebcdb5af"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "e491197d5e87ef8a97da6b342a17e1e9"
server_key_iv: "78af4312b223fb59646c7ee861979766"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.155851 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "a66f823c20b1493cbaa4747eebcdb5af"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "e491197d5e87ef8a97da6b342a17e1e9"
server_key_iv: "78af4312b223fb59646c7ee861979766"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.157265 2823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:50.158061 2838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.158239 2823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:50.158344 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "a66f823c20b1493cbaa4747eebcdb5af"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "e491197d5e87ef8a97da6b342a17e1e9"
server_key_iv: "78af4312b223fb59646c7ee861979766"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.158420 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:50.168424 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:50.170509 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:50.170677 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:50.170921 2823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:50.171227 2823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:50.171283 2823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.171325 2823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:50.171356 2823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.177681 2823 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:41737
I20260927 21:05:50.177803 2952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:41737 every 8 connection(s)
I20260927 21:05:50.178123 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:50.182078 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2823
I20260927 21:05:50.182190 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:50.182421 17147 external_mini_cluster.cc:1428] Setting key cebb335774adc5a0bdf0411e003dcbc3
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:50.189028 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.178890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:50953 (local address 127.16.190.254:45549)
0927 21:05:50.178959 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.178961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.179189 (+ 228us) server_negotiation.cc:407] Connection header received
0927 21:05:50.179754 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.179756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.179766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.179827 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.180889 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.181439 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.182865 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.183212 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.183272 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.185908 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.185910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.185910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:50.185919 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.187474 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.187962 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.187964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.187964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.188033 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.188293 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.188294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.188294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.188427 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.188506 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.188793 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.188925 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 21:05:50.189235 2955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.178880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45549 (local address 127.16.190.193:50953)
0927 21:05:50.179055 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.179092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.179624 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.179908 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.179912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.180522 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:50.180778 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.180786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.181521 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.181522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.182718 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.182721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.183296 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.183297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.183469 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.184041 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:50.184052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:50.185791 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:50.187582 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.187590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.187593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.187825 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.188102 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.188103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.188104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.188218 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.188497 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:50.188504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:50.188657 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.188773 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.188984 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260927 21:05:50.190022 2953 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45549
I20260927 21:05:50.190111 2953 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:50.190315 2953 heartbeater.cc:511] Master 127.16.190.254:45549 requested a full tablet report, sending...
I20260927 21:05:50.190963 2761 ts_manager.cc:194] Registered new tserver with Master: a66f823c20b1493cbaa4747eebcdb5af (127.16.190.193:41737)
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:05:50.191620 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:50953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:05:50.212257 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:50.294791 2958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:50.295033 2958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:50.295059 2958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:50.296350 2958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:50.296399 2958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:50.296427 2958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:50.297920 2958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:50.298148 2958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:50.298334 2958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:50.303793 2963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.303807 2964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.303767 2966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.304102 2958 server_base.cc:1034] running on GCE node
I20260927 21:05:50.304304 2958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:50.304530 2958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:50.305701 2958 hybrid_clock.cc:648] HybridClock initialized: now 1790543150305667 us; error 55 us; skew 500 ppm
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:50.308219 2958 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:05:50.308697 2958 webserver.cc:460] Webserver started at http://127.16.190.194:34655/ using document root <none> and password file <none>
I20260927 21:05:50.308971 2958 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:50.309020 2958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:50.309113 2958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:50.310153 2958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "7b5b3a7d749e4c98ac832d23954942b1"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "422e35bcefe3354714d80293484f45f6"
server_key_iv: "0e9f19f241ff113d3a24ec96aec047c2"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.310451 2958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "7b5b3a7d749e4c98ac832d23954942b1"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "422e35bcefe3354714d80293484f45f6"
server_key_iv: "0e9f19f241ff113d3a24ec96aec047c2"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.311756 2958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:50.312599 2973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.312911 2958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:50.312999 2958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "7b5b3a7d749e4c98ac832d23954942b1"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "422e35bcefe3354714d80293484f45f6"
server_key_iv: "0e9f19f241ff113d3a24ec96aec047c2"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.313062 2958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:50.334838 2958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:50.337132 2958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:50.337312 2958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:50.337571 2958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:50.337905 2958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:50.337962 2958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.337996 2958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:50.338029 2958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.344019 2958 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:35293
I20260927 21:05:50.344061 3087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:35293 every 8 connection(s)
I20260927 21:05:50.344399 2958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:50.347548 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 2958
I20260927 21:05:50.347695 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:50.347944 17147 external_mini_cluster.cc:1428] Setting key 68041f96c5c91f6d3ef228b962656fdc
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:50.354982 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.345126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:56989 (local address 127.16.190.254:45549)
0927 21:05:50.345185 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.345187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.345426 (+ 239us) server_negotiation.cc:407] Connection header received
0927 21:05:50.345919 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.345921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.345932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.346017 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.346993 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.347538 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.348977 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.349167 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.349226 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.351603 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.351605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.351605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:50.351614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.353515 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.353935 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.353936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.353937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.354005 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.354260 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.354261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.354262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.354434 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.354511 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.354770 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.354875 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 21:05:50.355026 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.345135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45549 (local address 127.16.190.194:56989)
0927 21:05:50.345296 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.345333 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.345811 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.346093 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.346097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.346667 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:50.346866 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.346871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.347656 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.347658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.348892 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.348896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.349261 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.349262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.349432 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.349999 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:50.350010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:50.351508 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:50.353612 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.353619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.353621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.353854 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.354078 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.354078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.354079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.354182 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.354507 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:50.354511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:50.354634 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.354722 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.354849 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260927 21:05:50.355535 3088 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45549
I20260927 21:05:50.355638 3088 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:50.355846 3088 heartbeater.cc:511] Master 127.16.190.254:45549 requested a full tablet report, sending...
I20260927 21:05:50.356247 2761 ts_manager.cc:194] Registered new tserver with Master: 7b5b3a7d749e4c98ac832d23954942b1 (127.16.190.194:35293)
I20260927 21:05:50.356647 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:56989
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:05:50.376248 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:50.460494 3093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:50.460872 3093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:50.460945 3093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:50.462261 3093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:50.462327 3093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:50.462373 3093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:50.463881 3093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:50.464177 3093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:50.464437 3093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:50.469587 3101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.469605 3098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.469600 3099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.469834 3093 server_base.cc:1034] running on GCE node
I20260927 21:05:50.470046 3093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:50.470319 3093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:50.471483 3093 hybrid_clock.cc:648] HybridClock initialized: now 1790543150471479 us; error 38 us; skew 500 ppm
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:50.474206 3093 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:05:50.474732 3093 webserver.cc:460] Webserver started at http://127.16.190.195:33773/ using document root <none> and password file <none>
I20260927 21:05:50.474989 3093 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:50.475062 3093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:50.475210 3093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:50.476328 3093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "459e23a05ea64aa59d6756128f3d033a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "f8d8e53cfd5517fce443a59fedf101fc"
server_key_iv: "383a208457e0b18c411f232b93969c39"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.476699 3093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "459e23a05ea64aa59d6756128f3d033a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "f8d8e53cfd5517fce443a59fedf101fc"
server_key_iv: "383a208457e0b18c411f232b93969c39"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.478094 3093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:50.478890 3108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.479094 3093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:05:50.479200 3093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "459e23a05ea64aa59d6756128f3d033a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "f8d8e53cfd5517fce443a59fedf101fc"
server_key_iv: "383a208457e0b18c411f232b93969c39"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.479300 3093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:50.497421 3093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:50.499591 3093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:50.499784 3093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:50.500036 3093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:50.500360 3093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:50.500415 3093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.500460 3093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:50.500494 3093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.506677 3093 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:46255
I20260927 21:05:50.506721 3221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:46255 every 8 connection(s)
I20260927 21:05:50.507047 3093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:50.511453 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3093
I20260927 21:05:50.511564 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:05:50.511763 17147 external_mini_cluster.cc:1428] Setting key d2f2cf16d77f3dd6ce698fb5c7db2bd6
Sep 27 21:05:50 dist-test-slave-6c7s krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:50.517728 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.507771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:43939 (local address 127.16.190.254:45549)
0927 21:05:50.507838 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.507840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.508032 (+ 192us) server_negotiation.cc:407] Connection header received
0927 21:05:50.508542 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.508544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.508554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.508615 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.509627 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.510129 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.511128 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.511388 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.511468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.514200 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.514201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.514201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:50.514208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.516148 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.516614 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.516615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.516616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.516675 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.517042 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.517043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.517043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.517192 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.517252 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.517575 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.517652 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:05:50.517928 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.507731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:45549 (local address 127.16.190.195:43939)
0927 21:05:50.507896 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.507938 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.508449 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.508671 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.508675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.509337 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:50.509540 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.509545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.510213 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.510214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.511037 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.511039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.511504 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.511505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.511707 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.512408 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:50.512427 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:50.514078 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:50.516248 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.516257 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.516259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.516539 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.516759 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:50.516760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:50.516760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:50.516971 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:50.517276 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:50.517285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:50.517435 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.517527 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.517662 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 21:05:50.518584 3222 heartbeater.cc:347] Connected to a master server at 127.16.190.254:45549
I20260927 21:05:50.518687 3222 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:50.518839 3222 heartbeater.cc:511] Master 127.16.190.254:45549 requested a full tablet report, sending...
I20260927 21:05:50.519213 2761 ts_manager.cc:194] Registered new tserver with Master: 459e23a05ea64aa59d6756128f3d033a (127.16.190.195:46255)
I20260927 21:05:50.519629 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:43939
I20260927 21:05:50.523792 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:50.531827 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.525218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.16.190.254:45549)
0927 21:05:50.525328 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.525330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.525339 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:05:50.525380 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.525381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.525389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.525437 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.526104 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.526637 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.527386 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.527599 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.527663 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.529098 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.529099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.529099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:50.529106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.530475 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.530875 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.530876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.530877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.530932 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:50.531167 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:50.531168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:50.531168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:50.531337 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.531442 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.531634 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.531693 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 21:05:50.533900 2761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:50.535543 2761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:50.542914 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:41737 (local address 127.0.0.1:37628)
0927 21:05:50.538814 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.538823 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.538884 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.539098 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.539099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.539101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:50.539153 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.539155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.540040 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.540041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.541433 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.541436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.542601 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.542602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.542681 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.542697 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.542774 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.542828 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":66,"threads_started":1}
I20260927 21:05:50.542949 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.16.190.193:41737)
0927 21:05:50.538765 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.538770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.538941 (+ 171us) server_negotiation.cc:407] Connection header received
0927 21:05:50.538986 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.538988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.539004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.539084 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:50.539208 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.539937 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.541541 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.542511 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.542609 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.542679 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.542799 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.542850 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260927 21:05:50.543179 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46255 (local address 127.0.0.1:52236)
0927 21:05:50.538303 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.538391 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.538456 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.538917 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.538921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.538927 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:50.539041 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.539044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.540523 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.540525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.541594 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.541596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.542738 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.542739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.542820 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.542824 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.542891 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.542937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
I20260927 21:05:50.543416 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.16.190.195:46255)
0927 21:05:50.538626 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.538631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.538643 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:50.538695 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.538698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.538715 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.538813 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:50.539140 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.540407 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.541778 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.542582 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.542650 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.542698 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.543266 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.543322 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260927 21:05:50.543639 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:35293 (local address 127.0.0.1:55228)
0927 21:05:50.538560 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.538569 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.538621 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.538917 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.538921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.538924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:50.539041 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.539043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.541662 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.541663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.542378 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.542380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.543391 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.543392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.543463 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.543468 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.543522 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.543576 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":79,"threads_started":1}
I20260927 21:05:50.543871 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.538319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.16.190.194:35293)
0927 21:05:50.538530 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.538535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.538646 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:05:50.538695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.538698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.538715 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.538813 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:50.539139 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.540403 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.542484 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.543237 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.543305 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.543356 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.543706 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.543765 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260927 21:05:50.544149 2887 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 (DEFAULT_TABLE table=test-table [id=6a934e37ad854e72aef294c227133bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:50.544414 3156 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 (DEFAULT_TABLE table=test-table [id=6a934e37ad854e72aef294c227133bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:50.544581 2887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9d272b6664d4e88d50a1ba65aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:50.544566 3021 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 (DEFAULT_TABLE table=test-table [id=6a934e37ad854e72aef294c227133bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:50.544744 3156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9d272b6664d4e88d50a1ba65aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:50.544878 3021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa9d272b6664d4e88d50a1ba65aebc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:50.547721 3239 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: Bootstrap starting.
I20260927 21:05:50.548058 3240 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Bootstrap starting.
I20260927 21:05:50.548346 3241 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Bootstrap starting.
I20260927 21:05:50.548609 3239 tablet_bootstrap.cc:654] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:50.548768 3240 tablet_bootstrap.cc:654] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:50.549014 3241 tablet_bootstrap.cc:654] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:50.549424 3239 log.cc:824] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:50.549463 3240 log.cc:824] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:50.549588 3241 log.cc:824] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:50.550318 3239 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: No bootstrap required, opened a new log
I20260927 21:05:50.550318 3241 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: No bootstrap required, opened a new log
I20260927 21:05:50.550396 3239 ts_tablet_manager.cc:1397] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 21:05:50.550396 3241 ts_tablet_manager.cc:1397] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:50.550515 3240 tablet_bootstrap.cc:492] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: No bootstrap required, opened a new log
I20260927 21:05:50.550588 3240 ts_tablet_manager.cc:1397] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 21:05:50.552052 3240 raft_consensus.cc:348] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552232 3240 raft_consensus.cc:374] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:50.552282 3240 raft_consensus.cc:729] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459e23a05ea64aa59d6756128f3d033a, State: Initialized, Role: FOLLOWER
I20260927 21:05:50.552366 3239 raft_consensus.cc:348] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552404 3240 consensus_queue.cc:260] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552366 3241 raft_consensus.cc:348] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552543 3241 raft_consensus.cc:374] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:50.552543 3239 raft_consensus.cc:374] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:50.552588 3239 raft_consensus.cc:729] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66f823c20b1493cbaa4747eebcdb5af, State: Initialized, Role: FOLLOWER
I20260927 21:05:50.552588 3241 raft_consensus.cc:729] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5b3a7d749e4c98ac832d23954942b1, State: Initialized, Role: FOLLOWER
I20260927 21:05:50.552697 3239 consensus_queue.cc:260] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552721 3241 consensus_queue.cc:260] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.552778 3240 ts_tablet_manager.cc:1428] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:05:50.552965 3222 heartbeater.cc:503] Master 127.16.190.254:45549 was elected leader, sending a full tablet report...
I20260927 21:05:50.552997 3241 ts_tablet_manager.cc:1428] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:50.552997 3239 ts_tablet_manager.cc:1428] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:05:50.553113 3088 heartbeater.cc:503] Master 127.16.190.254:45549 was elected leader, sending a full tablet report...
I20260927 21:05:50.553145 2953 heartbeater.cc:503] Master 127.16.190.254:45549 was elected leader, sending a full tablet report...
W20260927 21:05:50.595014 3089 tablet.cc:2369] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:05:50.600034 3246 raft_consensus.cc:482] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:50.600169 3246 raft_consensus.cc:504] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.600492 3246 leader_election.cc:288] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 459e23a05ea64aa59d6756128f3d033a (127.16.190.195:46255), a66f823c20b1493cbaa4747eebcdb5af (127.16.190.193:41737)
I20260927 21:05:50.603368 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.600800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:35029 (local address 127.16.190.195:46255)
0927 21:05:50.600873 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.600875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.600885 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:50.600916 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.600917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.600927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.600997 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:50.601175 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.601656 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.602610 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.602993 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.603089 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.603098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.603260 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.603304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:05:50.603368 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.600690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46255 (local address 127.16.190.194:35029)
0927 21:05:50.600752 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.600765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.600843 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.601019 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.601020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.601022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:50.601095 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.601096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.601746 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.601748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.602514 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.602516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.603087 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.603088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.603173 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.603176 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.603253 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.603295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":7}
I20260927 21:05:50.603533 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.600837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:41737 (local address 127.16.190.194:41679)
0927 21:05:50.601037 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.601048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.601126 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.601257 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.601258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.601259 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:50.601333 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.601335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.601957 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.601958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.602686 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.602688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.603272 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.603273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.603348 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.603351 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.603418 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.603456 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":134,"threads_started":1}
I20260927 21:05:50.603525 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.600913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:41679 (local address 127.16.190.193:41737)
0927 21:05:50.600969 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.600972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.601121 (+ 149us) server_negotiation.cc:407] Connection header received
0927 21:05:50.601199 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.601200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.601211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.601256 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:50.601403 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.601865 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.602790 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.603180 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.603272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.603280 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.603426 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.603464 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:05:50.604029 2907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9d272b6664d4e88d50a1ba65aebc3" candidate_uuid: "7b5b3a7d749e4c98ac832d23954942b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66f823c20b1493cbaa4747eebcdb5af" is_pre_election: true
I20260927 21:05:50.604192 2907 raft_consensus.cc:2454] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5b3a7d749e4c98ac832d23954942b1 in term 0.
I20260927 21:05:50.604449 2977 leader_election.cc:302] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5b3a7d749e4c98ac832d23954942b1, a66f823c20b1493cbaa4747eebcdb5af; no voters:
I20260927 21:05:50.604425 3176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9d272b6664d4e88d50a1ba65aebc3" candidate_uuid: "7b5b3a7d749e4c98ac832d23954942b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459e23a05ea64aa59d6756128f3d033a" is_pre_election: true
I20260927 21:05:50.604586 3176 raft_consensus.cc:2454] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5b3a7d749e4c98ac832d23954942b1 in term 0.
I20260927 21:05:50.604578 3246 raft_consensus.cc:2781] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:50.604630 3246 raft_consensus.cc:482] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:50.604656 3246 raft_consensus.cc:3037] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:50.605455 3246 raft_consensus.cc:504] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.605594 3246 leader_election.cc:288] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [CANDIDATE]: Term 1 election: Requested vote from peers 459e23a05ea64aa59d6756128f3d033a (127.16.190.195:46255), a66f823c20b1493cbaa4747eebcdb5af (127.16.190.193:41737)
I20260927 21:05:50.605779 3176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9d272b6664d4e88d50a1ba65aebc3" candidate_uuid: "7b5b3a7d749e4c98ac832d23954942b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "459e23a05ea64aa59d6756128f3d033a"
I20260927 21:05:50.605801 2907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa9d272b6664d4e88d50a1ba65aebc3" candidate_uuid: "7b5b3a7d749e4c98ac832d23954942b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a66f823c20b1493cbaa4747eebcdb5af"
I20260927 21:05:50.605860 3176 raft_consensus.cc:3037] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:50.605886 2907 raft_consensus.cc:3037] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:50.606601 2907 raft_consensus.cc:2454] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5b3a7d749e4c98ac832d23954942b1 in term 1.
I20260927 21:05:50.606611 3176 raft_consensus.cc:2454] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5b3a7d749e4c98ac832d23954942b1 in term 1.
I20260927 21:05:50.606814 2977 leader_election.cc:302] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5b3a7d749e4c98ac832d23954942b1, a66f823c20b1493cbaa4747eebcdb5af; no voters:
I20260927 21:05:50.606923 3246 raft_consensus.cc:2781] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:50.607084 3246 raft_consensus.cc:686] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5b3a7d749e4c98ac832d23954942b1, State: Running, Role: LEADER
I20260927 21:05:50.607187 3246 consensus_queue.cc:237] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } }
I20260927 21:05:50.608058 2758 catalog_manager.cc:5495] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5b3a7d749e4c98ac832d23954942b1 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "7b5b3a7d749e4c98ac832d23954942b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5b3a7d749e4c98ac832d23954942b1" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 35293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 } health_report { overall_health: UNKNOWN } } }
I20260927 21:05:50.616585 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.614159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.16.190.194:35293)
0927 21:05:50.614215 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.614217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.614225 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:50.614288 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.614289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.614303 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.614362 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:50.614472 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.614932 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.615697 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.615885 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.615941 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.616135 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:50.616263 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:50.616341 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.616441 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.616451 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:05:50.618466 3176 raft_consensus.cc:1261] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 1 FOLLOWER]: Refusing update from remote peer 7b5b3a7d749e4c98ac832d23954942b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:50.618466 2907 raft_consensus.cc:1261] T 1fa9d272b6664d4e88d50a1ba65aebc3 P a66f823c20b1493cbaa4747eebcdb5af [term 1 FOLLOWER]: Refusing update from remote peer 7b5b3a7d749e4c98ac832d23954942b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:50.618822 3246 consensus_queue.cc:1035] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "459e23a05ea64aa59d6756128f3d033a" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:50.618971 3249 consensus_queue.cc:1035] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a66f823c20b1493cbaa4747eebcdb5af" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:50.621636 3252 mvcc.cc:205] Tried to move back new op lower bound from 7334064744930045952 to 7334064744887730176. Current Snapshot: MvccSnapshot[applied={T|T < 7334064744930045952}]
I20260927 21:05:50.621738 3254 mvcc.cc:205] Tried to move back new op lower bound from 7334064744930045952 to 7334064744887730176. Current Snapshot: MvccSnapshot[applied={T|T < 7334064744930045952}]
I20260927 21:05:50.622287 3253 mvcc.cc:205] Tried to move back new op lower bound from 7334064744930045952 to 7334064744887730176. Current Snapshot: MvccSnapshot[applied={T|T < 7334064744930045952}]
I20260927 21:05:50.624202 2759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:50.624306 2759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:50.625131 2759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f16da5fcc540f2955ca109052e94ed: Sending DeleteTablet for 3 replicas of tablet 1fa9d272b6664d4e88d50a1ba65aebc3
I20260927 21:05:50.625490 3156 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:50 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:52236
I20260927 21:05:50.625546 2887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:50 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:37628
I20260927 21:05:50.625731 3021 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa9d272b6664d4e88d50a1ba65aebc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:50 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:55228
I20260927 21:05:50.625814 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2823
I20260927 21:05:50.625798 3261 tablet_replica.cc:330] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: stopping tablet replica
I20260927 21:05:50.626605 3261 raft_consensus.cc:2229] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:50.626807 3261 raft_consensus.cc:2258] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:50.627337 3261 ts_tablet_manager.cc:1905] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:50.628958 3263 tablet_replica.cc:330] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: stopping tablet replica
I20260927 21:05:50.628981 3261 ts_tablet_manager.cc:1918] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:50.629067 3263 raft_consensus.cc:2229] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:50.629170 3263 raft_consensus.cc:2258] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:50.629179 3261 log.cc:1196] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/1fa9d272b6664d4e88d50a1ba65aebc3
I20260927 21:05:50.629642 3261 ts_tablet_manager.cc:1939] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 459e23a05ea64aa59d6756128f3d033a: Deleting consensus metadata
I20260927 21:05:50.629645 3263 ts_tablet_manager.cc:1905] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:50.630151 2748 catalog_manager.cc:4841] TS 459e23a05ea64aa59d6756128f3d033a (127.16.190.195:46255): tablet 1fa9d272b6664d4e88d50a1ba65aebc3 (table test-table [id=6a934e37ad854e72aef294c227133bdb]) successfully deleted
I20260927 21:05:50.631354 3263 ts_tablet_manager.cc:1918] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:50.631474 3263 log.cc:1196] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/1fa9d272b6664d4e88d50a1ba65aebc3
I20260927 21:05:50.631723 3263 ts_tablet_manager.cc:1939] T 1fa9d272b6664d4e88d50a1ba65aebc3 P 7b5b3a7d749e4c98ac832d23954942b1: Deleting consensus metadata
I20260927 21:05:50.632196 2746 catalog_manager.cc:4841] TS 7b5b3a7d749e4c98ac832d23954942b1 (127.16.190.194:35293): tablet 1fa9d272b6664d4e88d50a1ba65aebc3 (table test-table [id=6a934e37ad854e72aef294c227133bdb]) successfully deleted
W20260927 21:05:50.632201 2748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:41737 (error 108)
I20260927 21:05:50.632586 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2958
I20260927 21:05:50.632738 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.632611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:41737 (local address 127.0.0.1:37634)
0927 21:05:50.632669 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.632703 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:41737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:05:50.633576 2748 catalog_manager.cc:4584] TS a66f823c20b1493cbaa4747eebcdb5af (127.16.190.193:41737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fa9d272b6664d4e88d50a1ba65aebc3: Network error: Client connection negotiation failed: client connection to 127.16.190.193:41737: connect: Connection refused (error 111)
I20260927 21:05:50.637575 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3093
I20260927 21:05:50.642248 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 2729
2026-09-27T21:05:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T21:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:50Z Disabled control of system clock
I20260927 21:05:50.669951 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38325
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:38325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:50.750969 3268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:50.751189 3268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:50.751210 3268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:50.752303 3268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:50.752342 3268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:50.752357 3268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:50.752369 3268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:50.752381 3268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:50.753762 3268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:38325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:50.753971 3268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:50.754257 3268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:50.759083 3276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.759093 3274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.759092 3273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.759411 3268 server_base.cc:1034] running on GCE node
I20260927 21:05:50.759639 3268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:50.759886 3268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:50.761050 3268 hybrid_clock.cc:648] HybridClock initialized: now 1790543150761039 us; error 27 us; skew 500 ppm
I20260927 21:05:50.767962 3268 webserver.cc:460] Webserver started at http://127.16.190.254:37783/ using document root <none> and password file <none>
I20260927 21:05:50.768313 3268 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:50.768395 3268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:50.768558 3268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:50.769764 3268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "f685395a7892401692257dcaab35de40"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "a7bd8c5eb6bc333fad83e863b50f5473"
server_key_iv: "3d1a14c114865099f42929a39b7764b3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.770140 3268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "f685395a7892401692257dcaab35de40"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "a7bd8c5eb6bc333fad83e863b50f5473"
server_key_iv: "3d1a14c114865099f42929a39b7764b3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.771550 3268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:50.772365 3282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.772584 3268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:50.772701 3268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "f685395a7892401692257dcaab35de40"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "a7bd8c5eb6bc333fad83e863b50f5473"
server_key_iv: "3d1a14c114865099f42929a39b7764b3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.772792 3268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:50.789836 3268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:50.790241 3268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:50.790413 3268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:50.794452 3268 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:38325
I20260927 21:05:50.794487 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:38325 every 8 connection(s)
I20260927 21:05:50.794874 3268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:50.795868 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:50.798259 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: Bootstrap starting.
I20260927 21:05:50.798868 3336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:50.799185 3336 log.cc:824] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:50.799922 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: No bootstrap required, opened a new log
I20260927 21:05:50.801614 3336 raft_consensus.cc:348] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } }
I20260927 21:05:50.801759 3336 raft_consensus.cc:374] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:50.801781 3336 raft_consensus.cc:729] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f685395a7892401692257dcaab35de40, State: Initialized, Role: FOLLOWER
I20260927 21:05:50.801930 3336 consensus_queue.cc:260] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } }
I20260927 21:05:50.802014 3336 raft_consensus.cc:388] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:50.802057 3336 raft_consensus.cc:482] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:50.802110 3336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:50.802654 3336 raft_consensus.cc:504] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } }
I20260927 21:05:50.802739 3336 leader_election.cc:302] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f685395a7892401692257dcaab35de40; no voters:
I20260927 21:05:50.802964 3336 leader_election.cc:288] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:50.803052 3339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:50.803237 3339 raft_consensus.cc:686] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [term 1 LEADER]: Becoming Leader. State: Replica: f685395a7892401692257dcaab35de40, State: Running, Role: LEADER
I20260927 21:05:50.803263 3336 sys_catalog.cc:564] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:50.803380 3339 consensus_queue.cc:237] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } }
I20260927 21:05:50.803836 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f685395a7892401692257dcaab35de40. Latest consensus state: current_term: 1 leader_uuid: "f685395a7892401692257dcaab35de40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } } }
I20260927 21:05:50.803941 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:50.803936 3340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f685395a7892401692257dcaab35de40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f685395a7892401692257dcaab35de40" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38325 } } }
I20260927 21:05:50.804214 3348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:50.804222 3340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:50.804546 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3268
I20260927 21:05:50.804683 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:50.804958 17147 external_mini_cluster.cc:1428] Setting key 8d97a6749c96191587a9c2499f257e59
I20260927 21:05:50.805629 3348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:50.807251 3348 catalog_manager.cc:1347] Generated new cluster ID: e97d8e510a804443bc75e1f3dea27402
I20260927 21:05:50.807310 3348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:50.811827 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.805547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.16.190.254:38325)
0927 21:05:50.805862 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.805872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.805983 (+ 111us) server_negotiation.cc:407] Connection header received
0927 21:05:50.806276 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.806287 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.806383 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.806538 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.806783 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.808324 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.809477 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.810171 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.810302 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.811066 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.811071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.811071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:50.811311 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.811354 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:50.811369 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.811436 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.811522 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.811605 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":81,"threads_started":1}
I20260927 21:05:50.819522 3348 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:50.820235 3348 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:50.827920 3348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: Generated new TSK 0
I20260927 21:05:50.828179 3348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:50.830919 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--builtin_ntp_servers=127.16.190.212:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:50.918612 3360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:50.918886 3360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:50.918915 3360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:50.920302 3360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:50.920360 3360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:50.920392 3360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:50.922089 3360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:50.922411 3360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:50.922770 3360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:50.929057 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:50.929080 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.929308 3360 server_base.cc:1034] running on GCE node
W20260927 21:05:50.929119 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:50.929738 3360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:50.930097 3360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:50.931285 3360 hybrid_clock.cc:648] HybridClock initialized: now 1790543150931268 us; error 36 us; skew 500 ppm
I20260927 21:05:50.932754 3360 webserver.cc:460] Webserver started at http://127.16.190.193:33655/ using document root <none> and password file <none>
I20260927 21:05:50.933051 3360 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:50.933110 3360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:50.933234 3360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:50.934664 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "a33e451084834049b88dff710ff0018a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "0db30c48db6857cc43f83c085d87f57b"
server_key_iv: "ba7fce68eb5ba9e44aa1595dddd60cc3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.935078 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "a33e451084834049b88dff710ff0018a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "0db30c48db6857cc43f83c085d87f57b"
server_key_iv: "ba7fce68eb5ba9e44aa1595dddd60cc3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.936728 3360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 21:05:50.937834 3374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.938054 3360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:50.938143 3360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "a33e451084834049b88dff710ff0018a"
format_stamp: "Formatted at 2026-09-27 21:05:50 on dist-test-slave-6c7s"
server_key: "0db30c48db6857cc43f83c085d87f57b"
server_key_iv: "ba7fce68eb5ba9e44aa1595dddd60cc3"
server_key_version: "encryptionkey@0"
I20260927 21:05:50.938218 3360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:50.949579 3360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:50.949887 3360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:50.949995 3360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:50.950177 3360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:50.950541 3360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:50.950577 3360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.950636 3360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:50.950661 3360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:50.957502 3360 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:42185
I20260927 21:05:50.957547 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:42185 every 8 connection(s)
I20260927 21:05:50.957967 3360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:50.964082 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.958900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:53311 (local address 127.16.190.254:38325)
0927 21:05:50.958969 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:50.958971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:50.959198 (+ 227us) server_negotiation.cc:407] Connection header received
0927 21:05:50.959987 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:50.959989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:50.960007 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:50.960080 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:50.960897 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.961456 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.962611 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.962832 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.962902 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.963816 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:50.963818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:50.963818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:50.963823 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:50.963839 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:50.963843 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:50.963893 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:50.963987 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:05:50.964014 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 21:05:50.964303 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:50.958823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38325 (local address 127.16.190.193:53311)
0927 21:05:50.959036 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:50.959077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:50.959720 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:50.960188 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:50.960192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:50.960200 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:50.960735 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.960740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.961572 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.961573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.962515 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:50.962519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:50.962934 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:50.962935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:50.963063 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:50.963661 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:50.963661 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:50.963692 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:50.963725 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:50.963907 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:50.963911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:50.963914 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:50.963974 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 21:05:50.964062 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260927 21:05:50.964903 3490 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38325
I20260927 21:05:50.964994 3490 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:50.965209 3490 heartbeater.cc:511] Master 127.16.190.254:38325 requested a full tablet report, sending...
I20260927 21:05:50.965821 3299 ts_manager.cc:194] Registered new tserver with Master: a33e451084834049b88dff710ff0018a (127.16.190.193:42185)
I20260927 21:05:50.966492 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3360
I20260927 21:05:50.966495 3299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.193:53311
I20260927 21:05:50.966624 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:50.966904 17147 external_mini_cluster.cc:1428] Setting key 27992662f1427de669d2162277addf51
I20260927 21:05:50.967947 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--builtin_ntp_servers=127.16.190.212:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:51.055497 3493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:51.055725 3493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:51.055749 3493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:51.057088 3493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:51.057143 3493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:51.057191 3493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:51.058684 3493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:51.058990 3493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:51.059338 3493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:51.064761 3501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:51.064781 3499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:51.064788 3498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:51.065001 3493 server_base.cc:1034] running on GCE node
I20260927 21:05:51.065222 3493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:51.065498 3493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:51.066666 3493 hybrid_clock.cc:648] HybridClock initialized: now 1790543151066668 us; error 46 us; skew 500 ppm
I20260927 21:05:51.068012 3493 webserver.cc:460] Webserver started at http://127.16.190.194:34189/ using document root <none> and password file <none>
I20260927 21:05:51.068285 3493 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:51.068360 3493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:51.068496 3493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:51.069667 3493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "65d23da442164cc9bc6e7a9b675405b5"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "9fd7d9cf075f435ac15f916f8d881bae"
server_key_iv: "69caf4545a86e056dd5f60c24f358ea4"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.070050 3493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "65d23da442164cc9bc6e7a9b675405b5"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "9fd7d9cf075f435ac15f916f8d881bae"
server_key_iv: "69caf4545a86e056dd5f60c24f358ea4"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.071393 3493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:51.072238 3507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.072434 3493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:51.072539 3493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "65d23da442164cc9bc6e7a9b675405b5"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "9fd7d9cf075f435ac15f916f8d881bae"
server_key_iv: "69caf4545a86e056dd5f60c24f358ea4"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.072618 3493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:51.082629 3493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:51.082978 3493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:51.083158 3493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:51.083407 3493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:51.083746 3493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:51.083801 3493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.083843 3493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:51.083879 3493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.089901 3493 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:44225
I20260927 21:05:51.089939 3621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:44225 every 8 connection(s)
I20260927 21:05:51.090337 3493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:51.093106 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3493
I20260927 21:05:51.093225 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:51.093472 17147 external_mini_cluster.cc:1428] Setting key b5fdf3e52d756970eb75bb45a7a23184
I20260927 21:05:51.094470 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--builtin_ntp_servers=127.16.190.212:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:05:51.096319 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.091112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:33359 (local address 127.16.190.254:38325)
0927 21:05:51.091188 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.091190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.091546 (+ 356us) server_negotiation.cc:407] Connection header received
0927 21:05:51.092037 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.092038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.092048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.092098 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:51.092750 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.093385 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.094791 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.094989 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.095047 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.096003 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:51.096005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:51.096005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:51.096012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:51.096029 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:51.096034 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:51.096095 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.096181 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.096213 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 21:05:51.096503 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.091097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38325 (local address 127.16.190.194:33359)
0927 21:05:51.091352 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.091449 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.091948 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.092155 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.092158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.092166 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:51.092666 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.092670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.093476 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.093477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.094714 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.094717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.095076 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.095078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.095249 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.095824 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:51.095825 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:51.095871 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:51.095905 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:51.096103 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:51.096107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:51.096110 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.096163 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.096251 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260927 21:05:51.097040 3622 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38325
I20260927 21:05:51.097147 3622 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:51.097361 3622 heartbeater.cc:511] Master 127.16.190.254:38325 requested a full tablet report, sending...
I20260927 21:05:51.097747 3299 ts_manager.cc:194] Registered new tserver with Master: 65d23da442164cc9bc6e7a9b675405b5 (127.16.190.194:44225)
I20260927 21:05:51.098268 3299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.194:33359
W20260927 21:05:51.178432 3625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:51.178649 3625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:51.178673 3625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:51.179800 3625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:51.179837 3625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:51.179863 3625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:51.181169 3625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:38325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:51.181375 3625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:51.181623 3625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:51.187254 3630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:51.187415 3631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:51.187345 3633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:51.187568 3625 server_base.cc:1034] running on GCE node
I20260927 21:05:51.187803 3625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:51.188081 3625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:51.189253 3625 hybrid_clock.cc:648] HybridClock initialized: now 1790543151189237 us; error 34 us; skew 500 ppm
I20260927 21:05:51.190656 3625 webserver.cc:460] Webserver started at http://127.16.190.195:43003/ using document root <none> and password file <none>
I20260927 21:05:51.190930 3625 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:51.191006 3625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:51.191162 3625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:51.192265 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "8f46138fd39f4d86a61ebee082b2d9f0"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "e7cc2b725018bd927c633005a52be57a"
server_key_iv: "f2e99e0755e959d50a697053df00ddc0"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.192639 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "8f46138fd39f4d86a61ebee082b2d9f0"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "e7cc2b725018bd927c633005a52be57a"
server_key_iv: "f2e99e0755e959d50a697053df00ddc0"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.194043 3625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:51.194830 3639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.195000 3625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 21:05:51.195097 3625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "8f46138fd39f4d86a61ebee082b2d9f0"
format_stamp: "Formatted at 2026-09-27 21:05:51 on dist-test-slave-6c7s"
server_key: "e7cc2b725018bd927c633005a52be57a"
server_key_iv: "f2e99e0755e959d50a697053df00ddc0"
server_key_version: "encryptionkey@0"
I20260927 21:05:51.195180 3625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:51.206944 3625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:51.207314 3625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:51.207484 3625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:51.207734 3625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:51.208084 3625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:51.208141 3625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.208184 3625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:51.208212 3625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:51.214370 3625 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:46765
I20260927 21:05:51.214424 3753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:46765 every 8 connection(s)
I20260927 21:05:51.214778 3625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:51.220192 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3625
I20260927 21:05:51.220328 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:05:51.220564 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.215574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:52783 (local address 127.16.190.254:38325)
0927 21:05:51.215648 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.215650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.215881 (+ 231us) server_negotiation.cc:407] Connection header received
0927 21:05:51.216388 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.216389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.216400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.216473 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:51.217204 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.217798 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.218835 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.219060 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.219127 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.220254 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:51.220256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:51.220256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:51.220263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:51.220278 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:51.220283 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:51.220328 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.220440 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.220472 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:05:51.220587 17147 external_mini_cluster.cc:1428] Setting key cde601587a3297b856491a2f8f01cf50
I20260927 21:05:51.220901 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.215566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:38325 (local address 127.16.190.195:52783)
0927 21:05:51.215748 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.215792 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.216296 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.216529 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.216532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.216540 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:51.217094 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.217100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.217903 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.217904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.218731 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.218733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.219174 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.219175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.219369 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.220079 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:51.220079 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:51.220112 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:51.220137 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:51.220328 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:51.220334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:51.220337 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.220437 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.220551 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260927 21:05:51.221290 3754 heartbeater.cc:347] Connected to a master server at 127.16.190.254:38325
I20260927 21:05:51.221415 3754 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:51.221621 3754 heartbeater.cc:511] Master 127.16.190.254:38325 requested a full tablet report, sending...
I20260927 21:05:51.222023 3299 ts_manager.cc:194] Registered new tserver with Master: 8f46138fd39f4d86a61ebee082b2d9f0 (127.16.190.195:46765)
I20260927 21:05:51.222358 3299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.195:52783
I20260927 21:05:51.233026 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:51.237843 3357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.234519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.16.190.254:38325)
0927 21:05:51.234585 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.234587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.234707 (+ 120us) server_negotiation.cc:407] Connection header received
0927 21:05:51.234755 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.234755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.234767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.234821 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:51.235034 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.235534 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.236407 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.236616 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.236684 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.237592 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:51.237594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:51.237594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:51.237598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:51.237612 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:51.237616 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:51.237697 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.237770 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.237781 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:05:51.239660 3299 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:51.241348 3299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:51.248139 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.243975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.16.190.195:46765)
0927 21:05:51.244550 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.244555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.244566 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:51.244607 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.244609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.244624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.244903 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:51.244963 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.246067 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247246 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.247798 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247875 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.247959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.248021 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.248060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":106,"threads_started":1}
I20260927 21:05:51.248147 3762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.243843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:46765 (local address 127.0.0.1:34212)
0927 21:05:51.243975 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.244018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.244080 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.244738 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.244739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.244741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:51.244788 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.244791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.246147 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.246148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.247155 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.247157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.247873 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.247941 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.247945 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.247998 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.248039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":53,"thread_start_us":100,"threads_started":1}
I20260927 21:05:51.248253 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.244186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.16.190.193:42185)
0927 21:05:51.244372 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.244378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.244390 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:51.244434 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.244437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.244454 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.244532 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:51.244782 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.245546 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.246887 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.247640 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247713 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.247801 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.248101 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.248149 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 21:05:51.248628 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.244335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.16.190.194:44225)
0927 21:05:51.244550 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.244555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.244566 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:51.244607 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.244609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.244624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.244711 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:51.244911 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.246040 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247403 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.248133 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.248199 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.248287 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.248516 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.248554 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260927 21:05:51.248565 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.244082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44225 (local address 127.0.0.1:47514)
0927 21:05:51.244362 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.244371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.244407 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.244774 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.244775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.244777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:51.244805 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.244833 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.246144 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.246145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.247295 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.247298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.248325 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.248326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.248409 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.248412 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.248468 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.248503 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":36,"thread_start_us":69,"threads_started":1}
I20260927 21:05:51.248049 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.243848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42185 (local address 127.0.0.1:34772)
0927 21:05:51.244200 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.244210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.244253 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.244623 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.244626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.244628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:51.244700 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.244702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.245645 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.245647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.246794 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.246796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.247736 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.247737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.247815 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.247832 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.247898 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.247950 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260927 21:05:51.249698 3687 tablet_service.cc:1467] Processing CreateTablet for tablet 88d181d9421748a2962832009e4d7c3e (DEFAULT_TABLE table=test-table [id=7326ad6958af4900810dde883c0e2c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:51.249691 3423 tablet_service.cc:1467] Processing CreateTablet for tablet 88d181d9421748a2962832009e4d7c3e (DEFAULT_TABLE table=test-table [id=7326ad6958af4900810dde883c0e2c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:51.249693 3555 tablet_service.cc:1467] Processing CreateTablet for tablet 88d181d9421748a2962832009e4d7c3e (DEFAULT_TABLE table=test-table [id=7326ad6958af4900810dde883c0e2c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:51.250074 3687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d181d9421748a2962832009e4d7c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:51.250075 3423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d181d9421748a2962832009e4d7c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:51.250075 3555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88d181d9421748a2962832009e4d7c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:51.252673 3768 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Bootstrap starting.
I20260927 21:05:51.253441 3768 tablet_bootstrap.cc:654] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:51.253417 3770 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: Bootstrap starting.
I20260927 21:05:51.253845 3768 log.cc:824] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:51.254092 3770 tablet_bootstrap.cc:654] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:51.254243 3769 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Bootstrap starting.
I20260927 21:05:51.254431 3770 log.cc:824] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:51.254956 3769 tablet_bootstrap.cc:654] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:51.255208 3770 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: No bootstrap required, opened a new log
I20260927 21:05:51.255299 3769 log.cc:824] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:51.255425 3770 ts_tablet_manager.cc:1397] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:51.255863 3768 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: No bootstrap required, opened a new log
I20260927 21:05:51.255949 3768 ts_tablet_manager.cc:1397] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 21:05:51.256058 3769 tablet_bootstrap.cc:492] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: No bootstrap required, opened a new log
I20260927 21:05:51.256129 3769 ts_tablet_manager.cc:1397] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:51.258316 3769 raft_consensus.cc:348] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.258296 3770 raft_consensus.cc:348] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.258486 3769 raft_consensus.cc:374] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:51.258486 3770 raft_consensus.cc:374] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:51.258531 3769 raft_consensus.cc:729] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d23da442164cc9bc6e7a9b675405b5, State: Initialized, Role: FOLLOWER
I20260927 21:05:51.258531 3770 raft_consensus.cc:729] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33e451084834049b88dff710ff0018a, State: Initialized, Role: FOLLOWER
I20260927 21:05:51.258671 3769 consensus_queue.cc:260] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.258694 3770 consensus_queue.cc:260] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.258811 3768 raft_consensus.cc:348] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.258970 3768 raft_consensus.cc:374] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:51.258986 3769 ts_tablet_manager.cc:1428] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:51.259001 3770 ts_tablet_manager.cc:1428] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 21:05:51.259016 3768 raft_consensus.cc:729] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f46138fd39f4d86a61ebee082b2d9f0, State: Initialized, Role: FOLLOWER
I20260927 21:05:51.259119 3490 heartbeater.cc:503] Master 127.16.190.254:38325 was elected leader, sending a full tablet report...
I20260927 21:05:51.259125 3768 consensus_queue.cc:260] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.259370 3768 ts_tablet_manager.cc:1428] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260927 21:05:51.259428 3622 heartbeater.cc:503] Master 127.16.190.254:38325 was elected leader, sending a full tablet report...
I20260927 21:05:51.259526 3754 heartbeater.cc:503] Master 127.16.190.254:38325 was elected leader, sending a full tablet report...
I20260927 21:05:51.292548 3776 raft_consensus.cc:482] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:51.292698 3776 raft_consensus.cc:504] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.293123 3776 leader_election.cc:288] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65d23da442164cc9bc6e7a9b675405b5 (127.16.190.194:44225), a33e451084834049b88dff710ff0018a (127.16.190.193:42185)
I20260927 21:05:51.296274 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.293416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:50729 (local address 127.16.190.193:42185)
0927 21:05:51.293465 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.293467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.293480 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:05:51.293514 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.293515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.293526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.293632 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:51.293803 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.294283 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.295528 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.295907 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.295993 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.296001 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.296156 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.296193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:05:51.296288 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.293447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:60871 (local address 127.16.190.194:44225)
0927 21:05:51.293493 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.293495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.293633 (+ 138us) server_negotiation.cc:407] Connection header received
0927 21:05:51.293691 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.293692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.293701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.293778 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:51.293951 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.294409 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.295553 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.295912 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.296005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.296013 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.296174 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.296216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:05:51.296326 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.293319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42185 (local address 127.16.190.195:50729)
0927 21:05:51.293382 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.293394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.293444 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.293638 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.293640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.293642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:51.293713 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.293720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.294376 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.294377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.295437 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.295440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.296002 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.296003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.296074 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.296077 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.296150 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.296192 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 21:05:51.296326 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.293319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44225 (local address 127.16.190.195:60871)
0927 21:05:51.293552 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.293559 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:51.293633 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:51.293780 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:51.293781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:51.293782 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:51.293860 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.293861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.294493 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.294494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.295465 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:51.295467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.296006 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.296007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:51.296085 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.296093 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 21:05:51.296156 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:05:51.296194 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":20,"thread_start_us":145,"threads_started":1}
I20260927 21:05:51.296765 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d181d9421748a2962832009e4d7c3e" candidate_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d23da442164cc9bc6e7a9b675405b5" is_pre_election: true
I20260927 21:05:51.296779 3443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d181d9421748a2962832009e4d7c3e" candidate_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33e451084834049b88dff710ff0018a" is_pre_election: true
I20260927 21:05:51.296957 3443 raft_consensus.cc:2454] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f46138fd39f4d86a61ebee082b2d9f0 in term 0.
I20260927 21:05:51.296957 3575 raft_consensus.cc:2454] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f46138fd39f4d86a61ebee082b2d9f0 in term 0.
I20260927 21:05:51.297199 3642 leader_election.cc:302] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f46138fd39f4d86a61ebee082b2d9f0, a33e451084834049b88dff710ff0018a; no voters:
I20260927 21:05:51.297355 3776 raft_consensus.cc:2781] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:51.297437 3776 raft_consensus.cc:482] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:51.297474 3776 raft_consensus.cc:3037] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:51.298221 3776 raft_consensus.cc:504] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.298378 3776 leader_election.cc:288] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [CANDIDATE]: Term 1 election: Requested vote from peers 65d23da442164cc9bc6e7a9b675405b5 (127.16.190.194:44225), a33e451084834049b88dff710ff0018a (127.16.190.193:42185)
I20260927 21:05:51.298700 3443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d181d9421748a2962832009e4d7c3e" candidate_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33e451084834049b88dff710ff0018a"
I20260927 21:05:51.298714 3575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88d181d9421748a2962832009e4d7c3e" candidate_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d23da442164cc9bc6e7a9b675405b5"
I20260927 21:05:51.298796 3575 raft_consensus.cc:3037] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:51.298796 3443 raft_consensus.cc:3037] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:51.299777 3443 raft_consensus.cc:2454] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f46138fd39f4d86a61ebee082b2d9f0 in term 1.
I20260927 21:05:51.299774 3575 raft_consensus.cc:2454] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f46138fd39f4d86a61ebee082b2d9f0 in term 1.
I20260927 21:05:51.299949 3643 leader_election.cc:302] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d23da442164cc9bc6e7a9b675405b5, 8f46138fd39f4d86a61ebee082b2d9f0; no voters:
I20260927 21:05:51.300088 3776 raft_consensus.cc:2781] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:51.300262 3776 raft_consensus.cc:686] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8f46138fd39f4d86a61ebee082b2d9f0, State: Running, Role: LEADER
I20260927 21:05:51.300369 3776 consensus_queue.cc:237] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } }
I20260927 21:05:51.301299 3297 catalog_manager.cc:5495] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f46138fd39f4d86a61ebee082b2d9f0 (127.16.190.195). New cstate: current_term: 1 leader_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f46138fd39f4d86a61ebee082b2d9f0" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 46765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 } health_report { overall_health: UNKNOWN } } }
I20260927 21:05:51.311838 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.309225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.16.190.195:46765)
0927 21:05:51.309298 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:51.309300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:51.309311 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:51.309388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:51.309389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:51.309415 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:51.309503 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:51.309656 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.310144 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.310967 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:51.311185 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:51.311281 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:51.311460 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:51.311569 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:51.311683 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:51.311713 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 21:05:51.311721 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:05:51.313654 3575 raft_consensus.cc:1261] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 1 FOLLOWER]: Refusing update from remote peer 8f46138fd39f4d86a61ebee082b2d9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:51.313686 3443 raft_consensus.cc:1261] T 88d181d9421748a2962832009e4d7c3e P a33e451084834049b88dff710ff0018a [term 1 FOLLOWER]: Refusing update from remote peer 8f46138fd39f4d86a61ebee082b2d9f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:51.313977 3776 consensus_queue.cc:1035] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d23da442164cc9bc6e7a9b675405b5" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:51.314113 3778 consensus_queue.cc:1035] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33e451084834049b88dff710ff0018a" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 42185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:51.316545 3781 mvcc.cc:205] Tried to move back new op lower bound from 7334064747778187264 to 7334064747727310848. Current Snapshot: MvccSnapshot[applied={T|T < 7334064747778187264}]
I20260927 21:05:51.317466 3783 mvcc.cc:205] Tried to move back new op lower bound from 7334064747778187264 to 7334064747727310848. Current Snapshot: MvccSnapshot[applied={T|T < 7334064747778187264}]
I20260927 21:05:51.318979 3297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:51.319083 3297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:51.319864 3297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f685395a7892401692257dcaab35de40: Sending DeleteTablet for 3 replicas of tablet 88d181d9421748a2962832009e4d7c3e
I20260927 21:05:51.320184 3687 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d181d9421748a2962832009e4d7c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:51 UTC) from {username='slave'} at 127.0.0.1:34212
I20260927 21:05:51.320230 3423 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d181d9421748a2962832009e4d7c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:51 UTC) from {username='slave'} at 127.0.0.1:34772
I20260927 21:05:51.320225 3555 tablet_service.cc:1514] Processing DeleteTablet for tablet 88d181d9421748a2962832009e4d7c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:51 UTC) from {username='slave'} at 127.0.0.1:47514
I20260927 21:05:51.320421 3791 tablet_replica.cc:330] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: stopping tablet replica
I20260927 21:05:51.320592 3791 raft_consensus.cc:2229] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:51.320698 3791 raft_consensus.cc:2258] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:51.320871 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3360
I20260927 21:05:51.323036 3793 tablet_replica.cc:330] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: stopping tablet replica
I20260927 21:05:51.323151 3793 raft_consensus.cc:2229] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:51.323293 3793 raft_consensus.cc:2258] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:51.325661 3791 ts_tablet_manager.cc:1905] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:51.325825 3793 ts_tablet_manager.cc:1905] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 21:05:51.327217 3285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:42185 (error 108)
I20260927 21:05:51.327525 3791 ts_tablet_manager.cc:1918] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:51.327606 3791 log.cc:1196] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/88d181d9421748a2962832009e4d7c3e
I20260927 21:05:51.327798 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3493
I20260927 21:05:51.327877 3791 ts_tablet_manager.cc:1939] T 88d181d9421748a2962832009e4d7c3e P 8f46138fd39f4d86a61ebee082b2d9f0: Deleting consensus metadata
I20260927 21:05:51.328254 3793 ts_tablet_manager.cc:1918] T 88d181d9421748a2962832009e4d7c3e P 65d23da442164cc9bc6e7a9b675405b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:51.328441 3286 catalog_manager.cc:4841] TS 8f46138fd39f4d86a61ebee082b2d9f0 (127.16.190.195:46765): tablet 88d181d9421748a2962832009e4d7c3e (table test-table [id=7326ad6958af4900810dde883c0e2c7f]) successfully deleted
I20260927 21:05:51.333491 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.333315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:44225 (local address 127.0.0.1:47524)
0927 21:05:51.333390 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.333446 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.194:44225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 21:05:51.333485 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3625
I20260927 21:05:51.333597 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:51.333492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:42185 (local address 127.0.0.1:34780)
0927 21:05:51.333542 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:51.333566 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:42185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:05:51.334381 3286 catalog_manager.cc:4584] TS 65d23da442164cc9bc6e7a9b675405b5 (127.16.190.194:44225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88d181d9421748a2962832009e4d7c3e: Network error: Client connection negotiation failed: client connection to 127.16.190.194:44225: connect: Connection refused (error 111)
I20260927 21:05:51.340410 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3268
2026-09-27T21:05:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (698 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:51 dist-test-slave-6c7s krb5kdc[3797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:51 dist-test-slave-6c7s krb5kdc[3797](info): set up 2 sockets
Sep 27 21:05:51 dist-test-slave-6c7s krb5kdc[3797](info): commencing operation
krb5kdc: starting...
W20260927 21:05:52.393654 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:05:52.520437 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37579
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:52.604471 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:52.604728 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:52.604768 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:52.605940 3810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:52.606016 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:52.606041 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:52.606072 3810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:52.606103 3810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:52.607616 3810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:37579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:52.607908 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:52.608139 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:52.613418 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:52.613418 3816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:52.613412 3818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:52.613818 3810 server_base.cc:1034] running on GCE node
I20260927 21:05:52.614082 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:52.614369 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:52.615562 3810 hybrid_clock.cc:648] HybridClock initialized: now 1790543152615547 us; error 29 us; skew 500 ppm
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:52.618224 3810 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:05:52.618680 3810 webserver.cc:460] Webserver started at http://127.16.190.254:38507/ using document root <none> and password file <none>
I20260927 21:05:52.618906 3810 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:52.618973 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:52.619123 3810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:52.620187 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "897eb979684945dbac9196490e6c52da"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "318e0a9e4f73f6b36ed3113025562208"
server_key_iv: "d0ddb078d19d058b3dc4f81a0dbff705"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.620563 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "897eb979684945dbac9196490e6c52da"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "318e0a9e4f73f6b36ed3113025562208"
server_key_iv: "d0ddb078d19d058b3dc4f81a0dbff705"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.622040 3810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:05:52.622828 3825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.623014 3810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:52.623113 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "897eb979684945dbac9196490e6c52da"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "318e0a9e4f73f6b36ed3113025562208"
server_key_iv: "d0ddb078d19d058b3dc4f81a0dbff705"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.623203 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:52.631299 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:52.633539 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:52.633706 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:52.637753 3810 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:37579
I20260927 21:05:52.637789 3877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:37579 every 8 connection(s)
I20260927 21:05:52.638137 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:52.639073 3878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:52.641618 3878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: Bootstrap starting.
I20260927 21:05:52.642261 3878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:52.642575 3878 log.cc:824] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:52.643352 3878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: No bootstrap required, opened a new log
I20260927 21:05:52.645144 3878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } }
I20260927 21:05:52.645323 3878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:52.645376 3878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897eb979684945dbac9196490e6c52da, State: Initialized, Role: FOLLOWER
I20260927 21:05:52.645538 3878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } }
I20260927 21:05:52.645612 3878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:52.645658 3878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:52.645714 3878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:52.646296 3878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } }
I20260927 21:05:52.646422 3878 leader_election.cc:302] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897eb979684945dbac9196490e6c52da; no voters:
I20260927 21:05:52.646577 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3810
I20260927 21:05:52.646626 3878 leader_election.cc:288] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:52.646653 3881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:52.646708 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:52.646847 3881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [term 1 LEADER]: Becoming Leader. State: Replica: 897eb979684945dbac9196490e6c52da, State: Running, Role: LEADER
I20260927 21:05:52.646924 17147 external_mini_cluster.cc:1428] Setting key 1ba420b46559dc9944f93b1a0f7c0822
I20260927 21:05:52.646900 3878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:52.646997 3881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } }
I20260927 21:05:52.647706 3882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897eb979684945dbac9196490e6c52da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } } }
I20260927 21:05:52.647791 3882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:52.647931 3883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897eb979684945dbac9196490e6c52da. Latest consensus state: current_term: 1 leader_uuid: "897eb979684945dbac9196490e6c52da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897eb979684945dbac9196490e6c52da" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 37579 } } }
I20260927 21:05:52.647986 3883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:52.648310 3892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:52.648952 3892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:52.650715 3892 catalog_manager.cc:1347] Generated new cluster ID: 41a3aa8f67fc4cdb83086da1b0d680d0
I20260927 21:05:52.650767 3892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:52.659979 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:52.647500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.16.190.254:37579)
0927 21:05:52.647688 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:52.647698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:52.647733 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:05:52.647858 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:52.647867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:52.647950 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:52.648143 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:52.649148 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.650405 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.651591 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.652196 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.652295 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:52.655292 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:52.655298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:52.655298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:52.655311 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:52.658022 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:52.658492 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:52.658495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:52.658495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:52.658618 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:52.658886 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:52.658886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:52.658887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:52.659195 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:52.659343 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:52.659490 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 21:05:52.659738 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260927 21:05:52.672385 3892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:52.673249 3892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:52.679085 3892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: Generated new TSK 0
I20260927 21:05:52.679320 3892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.190.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
Entry for principal kudu/127.16.190.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab.
I20260927 21:05:52.712183 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:52.797780 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:52.798007 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:52.798035 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:52.799234 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:52.799276 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:52.799304 3905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:52.800773 3905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.193.keytab
--principal=kudu/127.16.190.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:52.801067 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:52.801276 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:52.806694 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:52.806787 3905 server_base.cc:1034] running on GCE node
W20260927 21:05:52.806721 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:52.806732 3913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:52.807135 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:52.807415 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:52.808615 3905 hybrid_clock.cc:648] HybridClock initialized: now 1790543152808594 us; error 31 us; skew 500 ppm
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:52.811321 3905 init.cc:377] Logged in from keytab as kudu/127.16.190.193@KRBTEST.COM (short username kudu)
I20260927 21:05:52.811792 3905 webserver.cc:460] Webserver started at http://127.16.190.193:35781/ using document root <none> and password file <none>
I20260927 21:05:52.812088 3905 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:52.812165 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:52.812346 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:52.813587 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "3296efda59b74599b85f7724e9afcc7c"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "a31bfa1e5c414ca06d2211662e9fac75"
server_key_iv: "641226af5ddf6707048d74cbc26000f5"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.813989 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "3296efda59b74599b85f7724e9afcc7c"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "a31bfa1e5c414ca06d2211662e9fac75"
server_key_iv: "641226af5ddf6707048d74cbc26000f5"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.815413 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:05:52.816210 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.816401 3905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:52.816514 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "3296efda59b74599b85f7724e9afcc7c"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "a31bfa1e5c414ca06d2211662e9fac75"
server_key_iv: "641226af5ddf6707048d74cbc26000f5"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.816596 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:52.826529 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:52.829334 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:52.829520 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:52.829792 3905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:52.830132 3905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:52.830190 3905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.830240 3905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:52.830267 3905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.836460 3905 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:34813
I20260927 21:05:52.836536 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:34813 every 8 connection(s)
I20260927 21:05:52.836933 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:52.837361 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 3905
I20260927 21:05:52.837483 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:52.837720 17147 external_mini_cluster.cc:1428] Setting key 8931d034766b668a47083b4c04b5865f
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.193@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.194@KRBTEST.COM; defaulting to no policy
I20260927 21:05:52.847967 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:52.837747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:36751 (local address 127.16.190.254:37579)
0927 21:05:52.837826 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:52.837830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:52.838185 (+ 355us) server_negotiation.cc:407] Connection header received
0927 21:05:52.838742 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:52.838743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:52.838757 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:52.838824 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:52.839973 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.840538 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.841635 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.842083 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.842164 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:52.844703 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:52.844705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:52.844706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:52.844716 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:52.846549 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:52.847030 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:52.847032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:52.847033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:52.847104 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:52.847357 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:52.847358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:52.847359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:52.847498 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:52.847559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:52.847820 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 21:05:52.847895 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:05:52.848155 4036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:52.837689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37579 (local address 127.16.190.193:36751)
0927 21:05:52.838047 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:52.838094 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:52.838635 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:52.838907 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:52.838911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:52.839548 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:52.839854 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:52.839860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.840657 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.840658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:52.841544 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:52.841546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:52.842179 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:52.842179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:52.842289 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:52.842884 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:52.842893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:52.844602 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:52.846634 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:52.846640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:52.846642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:52.846858 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:52.847179 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:52.847180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:52.847180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:52.847282 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:52.847560 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:52.847565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:52.847683 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 21:05:52.847767 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 21:05:52.847906 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":94,"threads_started":1}
I20260927 21:05:52.848704 4034 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37579
I20260927 21:05:52.848801 4034 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:52.849004 4034 heartbeater.cc:511] Master 127.16.190.254:37579 requested a full tablet report, sending...
I20260927 21:05:52.849606 3842 ts_manager.cc:194] Registered new tserver with Master: 3296efda59b74599b85f7724e9afcc7c (127.16.190.193:34813)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.194@KRBTEST.COM" created.
I20260927 21:05:52.850318 3842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.193@KRBTEST.COM'} at 127.16.190.193:36751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
Entry for principal kudu/127.16.190.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab.
I20260927 21:05:52.867839 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:52.953055 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:52.953271 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:52.953295 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:52.954393 4039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:52.954428 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:52.954454 4039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:52.955819 4039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.194.keytab
--principal=kudu/127.16.190.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:52.956023 4039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:52.956213 4039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:52.962092 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:52.962085 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:52.962105 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:52.962390 4039 server_base.cc:1034] running on GCE node
I20260927 21:05:52.962634 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:52.962898 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:52.964075 4039 hybrid_clock.cc:648] HybridClock initialized: now 1790543152964068 us; error 36 us; skew 500 ppm
Sep 27 21:05:52 dist-test-slave-6c7s krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:52.966807 4039 init.cc:377] Logged in from keytab as kudu/127.16.190.194@KRBTEST.COM (short username kudu)
I20260927 21:05:52.967262 4039 webserver.cc:460] Webserver started at http://127.16.190.194:44265/ using document root <none> and password file <none>
I20260927 21:05:52.967502 4039 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:52.967571 4039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:52.967732 4039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:52.968933 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "9a2e4cc9b9f2af5e51f56bec272eb3cd"
server_key_iv: "96cd8b79c6d0e00e77659c9f671ddc14"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.969296 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "9a2e4cc9b9f2af5e51f56bec272eb3cd"
server_key_iv: "96cd8b79c6d0e00e77659c9f671ddc14"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.970745 4039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:05:52.971593 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.971800 4039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:05:52.971918 4039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4"
format_stamp: "Formatted at 2026-09-27 21:05:52 on dist-test-slave-6c7s"
server_key: "9a2e4cc9b9f2af5e51f56bec272eb3cd"
server_key_iv: "96cd8b79c6d0e00e77659c9f671ddc14"
server_key_version: "encryptionkey@0"
I20260927 21:05:52.971997 4039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:52.992408 4039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:52.994947 4039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:52.995117 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:52.995388 4039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:52.995708 4039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:52.995761 4039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:52.995807 4039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:52.995837 4039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.001933 4039 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:34539
I20260927 21:05:53.002019 4169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:34539 every 8 connection(s)
I20260927 21:05:53.002358 4039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:53.003353 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4039
I20260927 21:05:53.003484 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:53.003701 17147 external_mini_cluster.cc:1428] Setting key b00466e393d885747bdf41c60d0499e7
Sep 27 21:05:53 dist-test-slave-6c7s krb5kdc[3797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543152, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.194@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.195@KRBTEST.COM; defaulting to no policy
I20260927 21:05:53.013749 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.003261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:56167 (local address 127.16.190.254:37579)
0927 21:05:53.003329 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.003331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.003470 (+ 139us) server_negotiation.cc:407] Connection header received
0927 21:05:53.004106 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.004107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.004120 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.004259 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.005345 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.006117 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.007507 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.007737 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.007829 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.010605 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.010607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.010607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:53.010616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.012112 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.012619 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.012621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.012621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.012685 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.012981 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.012983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.012983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.013109 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.013196 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.013558 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.013652 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 21:05:53.013957 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.003147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37579 (local address 127.16.190.194:56167)
0927 21:05:53.003320 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.003365 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.003984 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.004342 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.004346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.005024 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:53.005240 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.005245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.006292 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.006294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.007415 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.007418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.007842 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.007843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.007973 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.008543 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:53.008552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:53.010498 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:53.012229 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:53.012237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:53.012239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:53.012540 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:53.012753 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:53.012754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:53.012754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:53.012911 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:53.013180 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:53.013187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:53.013389 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.013521 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.013706 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260927 21:05:53.014513 4170 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37579
I20260927 21:05:53.014617 4170 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:53.014775 4170 heartbeater.cc:511] Master 127.16.190.254:37579 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.195@KRBTEST.COM" created.
I20260927 21:05:53.015194 3842 ts_manager.cc:194] Registered new tserver with Master: b6a543c4fa5b4a7c86bfa8126dac79b4 (127.16.190.194:34539)
I20260927 21:05:53.015710 3842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.194@KRBTEST.COM'} at 127.16.190.194:56167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
Entry for principal kudu/127.16.190.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab.
I20260927 21:05:53.033067 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:53.116088 4175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:53.116334 4175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:53.116362 4175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:53.117669 4175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:53.117718 4175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:53.117746 4175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:53.119251 4175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.195.keytab
--principal=kudu/127.16.190.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:37579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:53.119529 4175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:53.119786 4175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:53.125129 4181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:53.125137 4180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.125211 4175 server_base.cc:1034] running on GCE node
W20260927 21:05:53.125139 4183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.125627 4175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:53.125939 4175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:53.127128 4175 hybrid_clock.cc:648] HybridClock initialized: now 1790543153127114 us; error 27 us; skew 500 ppm
Sep 27 21:05:53 dist-test-slave-6c7s krb5kdc[3797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543153, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:53.129884 4175 init.cc:377] Logged in from keytab as kudu/127.16.190.195@KRBTEST.COM (short username kudu)
I20260927 21:05:53.130379 4175 webserver.cc:460] Webserver started at http://127.16.190.195:35007/ using document root <none> and password file <none>
I20260927 21:05:53.130635 4175 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:53.130681 4175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:53.130846 4175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:53.131923 4175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "1d891f80b2b34f5aa8e026038329c960"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "293c5fa56fe1c9a6c7b771e37b7e1405"
server_key_iv: "7da7a3c29030cce46a572ca927969430"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.132290 4175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "1d891f80b2b34f5aa8e026038329c960"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "293c5fa56fe1c9a6c7b771e37b7e1405"
server_key_iv: "7da7a3c29030cce46a572ca927969430"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.133756 4175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:53.134492 4190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.134717 4175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:53.134824 4175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "1d891f80b2b34f5aa8e026038329c960"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "293c5fa56fe1c9a6c7b771e37b7e1405"
server_key_iv: "7da7a3c29030cce46a572ca927969430"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.134900 4175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:53.160434 4175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:53.162957 4175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:53.163087 4175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:53.163328 4175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:53.163604 4175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:53.163633 4175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.163656 4175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:53.163672 4175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.169579 4175 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:36215
I20260927 21:05:53.169617 4303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:36215 every 8 connection(s)
I20260927 21:05:53.169968 4175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
Sep 27 21:05:53 dist-test-slave-6c7s krb5kdc[3797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543153, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.195@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:05:53.178390 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4175
I20260927 21:05:53.178483 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:05:53.178786 17147 external_mini_cluster.cc:1428] Setting key 0316758f45cbe38ced9d5bc951543e2f
I20260927 21:05:53.180662 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.170730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:45725 (local address 127.16.190.254:37579)
0927 21:05:53.170797 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.170800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.171012 (+ 212us) server_negotiation.cc:407] Connection header received
0927 21:05:53.171500 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.171501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.171511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.171574 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.172583 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.173180 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.174228 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.174423 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.174491 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.177265 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.177266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.177267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:53.177276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.178786 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.179488 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.179489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.179490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.179558 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.179963 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.179964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.179964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.180099 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.180173 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.180482 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.180570 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:05:53.180902 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.170700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:37579 (local address 127.16.190.195:45725)
0927 21:05:53.170881 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.170923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.171418 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.171643 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.171647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.172227 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:53.172463 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.172469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.173262 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.173263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.174125 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.174128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.174519 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.174520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.174635 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.175331 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:05:53.175341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:53.177138 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:53.178973 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:53.178985 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:53.179010 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:53.179384 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:53.179662 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:05:53.179663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:05:53.179664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:05:53.179856 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:05:53.180176 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:53.180184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:53.180334 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.180445 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.180623 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260927 21:05:53.181348 4304 heartbeater.cc:347] Connected to a master server at 127.16.190.254:37579
I20260927 21:05:53.181463 4304 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:53.181613 4304 heartbeater.cc:511] Master 127.16.190.254:37579 requested a full tablet report, sending...
I20260927 21:05:53.181952 3842 ts_manager.cc:194] Registered new tserver with Master: 1d891f80b2b34f5aa8e026038329c960 (127.16.190.195:36215)
I20260927 21:05:53.182372 3842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.190.195@KRBTEST.COM'} at 127.16.190.195:45725
I20260927 21:05:53.190907 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:53.199635 3889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.193680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.16.190.254:37579)
0927 21:05:53.193764 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.193766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.193778 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:53.193842 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.193843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.193853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.193920 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.194525 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.195038 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.195759 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.195948 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.196018 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.197319 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.197320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.197320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:53.197328 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.198554 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.198855 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.198856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.198856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.198906 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:53.199141 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:53.199141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:53.199142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:53.199289 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.199366 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.199501 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.199565 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:05:53.201452 3842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:53.203191 3842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:53.209915 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.205672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:36215 (local address 127.0.0.1:34964)
0927 21:05:53.205830 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.205851 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.205917 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.206270 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.206274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.206276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.206352 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.206354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207412 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.207413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.208831 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.208833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209652 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.209727 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.209731 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.209763 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.209814 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260927 21:05:53.209923 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.205665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:34813 (local address 127.0.0.1:55116)
0927 21:05:53.205807 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.205851 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.205917 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.206393 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.206395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.206397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.206457 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.206459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207842 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.207843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.208930 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.208931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209580 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.209657 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.209670 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.209738 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.209797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260927 21:05:53.210012 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.206038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.16.190.193:34813)
0927 21:05:53.206224 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.206228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.206238 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:53.206277 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.206279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.206293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.206532 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.206608 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207733 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209007 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209488 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209579 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.209639 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.209862 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.209920 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260927 21:05:53.210279 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.205802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.16.190.195:36215)
0927 21:05:53.206079 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.206083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.206091 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:53.206142 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.206144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.206162 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.206260 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.206439 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207279 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.208898 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209561 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.210047 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.210118 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.210147 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.210190 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260927 21:05:53.210294 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.205876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:34539 (local address 127.0.0.1:50194)
0927 21:05:53.206380 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.206390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.206437 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.206647 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.206648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.206650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.206692 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.206693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207894 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.207895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.208635 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.208636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209666 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.209729 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.209732 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.209782 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.209818 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260927 21:05:53.210616 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.206079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.16.190.194:34539)
0927 21:05:53.206245 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.206250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.206502 (+ 252us) server_negotiation.cc:407] Connection header received
0927 21:05:53.206548 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.206550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.206564 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.206637 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.206746 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.207795 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.208734 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.209500 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.209577 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.209639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.209861 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.209906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260927 21:05:53.210855 3968 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 (DEFAULT_TABLE table=test-table [id=57e41a2f199e4ad79101a34a9cecf03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.210856 4238 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 (DEFAULT_TABLE table=test-table [id=57e41a2f199e4ad79101a34a9cecf03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.211256 3968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a9f1a618149d2bdc6d95bab498088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.211256 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a9f1a618149d2bdc6d95bab498088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.211294 4103 tablet_service.cc:1467] Processing CreateTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 (DEFAULT_TABLE table=test-table [id=57e41a2f199e4ad79101a34a9cecf03f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.211670 4103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9a9f1a618149d2bdc6d95bab498088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.214470 4321 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Bootstrap starting.
I20260927 21:05:53.214887 4323 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: Bootstrap starting.
I20260927 21:05:53.214876 4322 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Bootstrap starting.
I20260927 21:05:53.215233 4321 tablet_bootstrap.cc:654] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.215549 4322 tablet_bootstrap.cc:654] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.215549 4323 tablet_bootstrap.cc:654] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.215849 4321 log.cc:824] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.216184 4322 log.cc:824] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.216184 4323 log.cc:824] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.216779 4321 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: No bootstrap required, opened a new log
I20260927 21:05:53.216879 4321 ts_tablet_manager.cc:1397] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:53.216934 4323 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: No bootstrap required, opened a new log
I20260927 21:05:53.217003 4323 ts_tablet_manager.cc:1397] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:53.217116 4322 tablet_bootstrap.cc:492] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: No bootstrap required, opened a new log
I20260927 21:05:53.217222 4322 ts_tablet_manager.cc:1397] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:53.218601 4322 raft_consensus.cc:348] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.218776 4322 raft_consensus.cc:374] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.218822 4322 raft_consensus.cc:729] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d891f80b2b34f5aa8e026038329c960, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.218935 4322 consensus_queue.cc:260] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.218957 4323 raft_consensus.cc:348] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.218957 4321 raft_consensus.cc:348] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.219139 4323 raft_consensus.cc:374] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.219139 4321 raft_consensus.cc:374] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.219223 4321 raft_consensus.cc:729] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a543c4fa5b4a7c86bfa8126dac79b4, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.219223 4323 raft_consensus.cc:729] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3296efda59b74599b85f7724e9afcc7c, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.219269 4322 ts_tablet_manager.cc:1428] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 21:05:53.219354 4323 consensus_queue.cc:260] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.219435 4304 heartbeater.cc:503] Master 127.16.190.254:37579 was elected leader, sending a full tablet report...
I20260927 21:05:53.219642 4323 ts_tablet_manager.cc:1428] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:05:53.219673 4321 consensus_queue.cc:260] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.219789 4034 heartbeater.cc:503] Master 127.16.190.254:37579 was elected leader, sending a full tablet report...
I20260927 21:05:53.219955 4321 ts_tablet_manager.cc:1428] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 21:05:53.220079 4170 heartbeater.cc:503] Master 127.16.190.254:37579 was elected leader, sending a full tablet report...
W20260927 21:05:53.253044 4171 tablet.cc:2369] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 21:05:53.267722 4329 raft_consensus.cc:482] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:53.267899 4329 raft_consensus.cc:504] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.268286 4329 leader_election.cc:288] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d891f80b2b34f5aa8e026038329c960 (127.16.190.195:36215), 3296efda59b74599b85f7724e9afcc7c (127.16.190.193:34813)
I20260927 21:05:53.272390 4319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.268634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:44841 (local address 127.16.190.193:34813)
0927 21:05:53.268681 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.268684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.268914 (+ 230us) server_negotiation.cc:407] Connection header received
0927 21:05:53.268960 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.268961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.268974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.269031 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.269173 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.269847 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.271376 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.271978 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.272047 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.272055 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.272275 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.272314 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:05:53.272423 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.268556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:48573 (local address 127.16.190.195:36215)
0927 21:05:53.268633 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.268635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.268646 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:05:53.268679 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.268680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.268691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.268753 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.268974 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.269777 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.271383 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.271965 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.272046 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.272055 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.272275 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.272315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:05:53.272456 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.268485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:36215 (local address 127.16.190.194:48573)
0927 21:05:53.268547 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.268562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.268621 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.268762 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.268763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.268765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.268893 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.268895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.269861 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.269861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.271277 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.271280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.272087 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.272089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.272185 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.272190 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.272258 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.272320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":118}
I20260927 21:05:53.272464 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.268486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:34813 (local address 127.16.190.194:44841)
0927 21:05:53.268789 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.268829 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.268896 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.269044 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.269045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.269046 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.269103 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.269104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.269921 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.269922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.271277 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.271280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.272087 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.272089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.272186 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.272190 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.272258 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.272319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":149,"thread_start_us":129,"threads_started":1}
I20260927 21:05:53.272847 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a9f1a618149d2bdc6d95bab498088" candidate_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d891f80b2b34f5aa8e026038329c960" is_pre_election: true
I20260927 21:05:53.272976 3988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a9f1a618149d2bdc6d95bab498088" candidate_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296efda59b74599b85f7724e9afcc7c" is_pre_election: true
I20260927 21:05:53.273051 4258 raft_consensus.cc:2454] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a543c4fa5b4a7c86bfa8126dac79b4 in term 0.
I20260927 21:05:53.273104 3988 raft_consensus.cc:2454] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6a543c4fa5b4a7c86bfa8126dac79b4 in term 0.
I20260927 21:05:53.273324 4056 leader_election.cc:302] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d891f80b2b34f5aa8e026038329c960, b6a543c4fa5b4a7c86bfa8126dac79b4; no voters:
I20260927 21:05:53.273461 4329 raft_consensus.cc:2781] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:53.273514 4329 raft_consensus.cc:482] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:53.273548 4329 raft_consensus.cc:3037] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.274279 4329 raft_consensus.cc:504] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.274441 4329 leader_election.cc:288] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1d891f80b2b34f5aa8e026038329c960 (127.16.190.195:36215), 3296efda59b74599b85f7724e9afcc7c (127.16.190.193:34813)
I20260927 21:05:53.274600 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a9f1a618149d2bdc6d95bab498088" candidate_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d891f80b2b34f5aa8e026038329c960"
I20260927 21:05:53.274601 3988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9a9f1a618149d2bdc6d95bab498088" candidate_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3296efda59b74599b85f7724e9afcc7c"
I20260927 21:05:53.274684 4258 raft_consensus.cc:3037] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.274708 3988 raft_consensus.cc:3037] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.275349 4258 raft_consensus.cc:2454] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a543c4fa5b4a7c86bfa8126dac79b4 in term 1.
I20260927 21:05:53.275519 4056 leader_election.cc:302] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d891f80b2b34f5aa8e026038329c960, b6a543c4fa5b4a7c86bfa8126dac79b4; no voters:
I20260927 21:05:53.275626 4329 raft_consensus.cc:2781] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:53.275676 3988 raft_consensus.cc:2454] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6a543c4fa5b4a7c86bfa8126dac79b4 in term 1.
I20260927 21:05:53.275825 4329 raft_consensus.cc:686] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 1 LEADER]: Becoming Leader. State: Replica: b6a543c4fa5b4a7c86bfa8126dac79b4, State: Running, Role: LEADER
I20260927 21:05:53.275933 4329 consensus_queue.cc:237] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } }
I20260927 21:05:53.276877 3841 catalog_manager.cc:5495] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6a543c4fa5b4a7c86bfa8126dac79b4 (127.16.190.194). New cstate: current_term: 1 leader_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6a543c4fa5b4a7c86bfa8126dac79b4" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 34539 } health_report { overall_health: HEALTHY } } }
I20260927 21:05:53.286806 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.284248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.16.190.194:34539)
0927 21:05:53.284318 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.284320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.284328 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:53.284361 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.284363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.284390 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.284451 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:53.284621 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.285285 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.286064 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.286248 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.286304 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.286506 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:53.286618 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:53.286697 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.286716 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.286744 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 21:05:53.288789 4258 raft_consensus.cc:1261] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 1 FOLLOWER]: Refusing update from remote peer b6a543c4fa5b4a7c86bfa8126dac79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:53.288807 3988 raft_consensus.cc:1261] T cf9a9f1a618149d2bdc6d95bab498088 P 3296efda59b74599b85f7724e9afcc7c [term 1 FOLLOWER]: Refusing update from remote peer b6a543c4fa5b4a7c86bfa8126dac79b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:53.289263 4329 consensus_queue.cc:1035] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3296efda59b74599b85f7724e9afcc7c" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:53.289451 4331 consensus_queue.cc:1035] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d891f80b2b34f5aa8e026038329c960" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:53.291857 4334 mvcc.cc:205] Tried to move back new op lower bound from 7334064755867639808 to 7334064755818917888. Current Snapshot: MvccSnapshot[applied={T|T < 7334064755867639808}]
I20260927 21:05:53.292445 4335 mvcc.cc:205] Tried to move back new op lower bound from 7334064755867639808 to 7334064755818917888. Current Snapshot: MvccSnapshot[applied={T|T < 7334064755867639808 or (T in {7334064755867639808})}]
I20260927 21:05:53.292637 4336 mvcc.cc:205] Tried to move back new op lower bound from 7334064755867639808 to 7334064755818917888. Current Snapshot: MvccSnapshot[applied={T|T < 7334064755867639808 or (T in {7334064755867639808})}]
I20260927 21:05:53.295456 3842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:53.295562 3842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:53.296427 3842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897eb979684945dbac9196490e6c52da: Sending DeleteTablet for 3 replicas of tablet cf9a9f1a618149d2bdc6d95bab498088
I20260927 21:05:53.296844 4238 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:34964
I20260927 21:05:53.297104 4344 tablet_replica.cc:330] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: stopping tablet replica
I20260927 21:05:53.297127 3968 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:55116
I20260927 21:05:53.297235 4344 raft_consensus.cc:2229] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:53.297360 4344 raft_consensus.cc:2258] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:53.297561 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3905
I20260927 21:05:53.297588 4103 tablet_service.cc:1514] Processing DeleteTablet for tablet cf9a9f1a618149d2bdc6d95bab498088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:53 UTC) from {username='kudu', principal='kudu/127.16.190.254@KRBTEST.COM'} at 127.0.0.1:50194
I20260927 21:05:53.297833 4344 ts_tablet_manager.cc:1905] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:53.300951 4346 tablet_replica.cc:330] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: stopping tablet replica
I20260927 21:05:53.301132 4346 raft_consensus.cc:2229] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 21:05:53.301131 4344 ts_tablet_manager.cc:1918] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:53.301205 4344 log.cc:1196] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/cf9a9f1a618149d2bdc6d95bab498088
I20260927 21:05:53.301431 4346 raft_consensus.cc:2258] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:53.301479 4344 ts_tablet_manager.cc:1939] T cf9a9f1a618149d2bdc6d95bab498088 P 1d891f80b2b34f5aa8e026038329c960: Deleting consensus metadata
I20260927 21:05:53.302009 4346 ts_tablet_manager.cc:1905] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:53.302232 3826 catalog_manager.cc:4841] TS 1d891f80b2b34f5aa8e026038329c960 (127.16.190.195:36215): tablet cf9a9f1a618149d2bdc6d95bab498088 (table test-table [id=57e41a2f199e4ad79101a34a9cecf03f]) successfully deleted
I20260927 21:05:53.303800 4346 ts_tablet_manager.cc:1918] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:53.303884 4346 log.cc:1196] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/cf9a9f1a618149d2bdc6d95bab498088
I20260927 21:05:53.304175 4346 ts_tablet_manager.cc:1939] T cf9a9f1a618149d2bdc6d95bab498088 P b6a543c4fa5b4a7c86bfa8126dac79b4: Deleting consensus metadata
W20260927 21:05:53.304672 3829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.190.193:34813 (error 108)
I20260927 21:05:53.304960 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4039
I20260927 21:05:53.305011 3826 catalog_manager.cc:4841] TS b6a543c4fa5b4a7c86bfa8126dac79b4 (127.16.190.194:34539): tablet cf9a9f1a618149d2bdc6d95bab498088 (table test-table [id=57e41a2f199e4ad79101a34a9cecf03f]) successfully deleted
I20260927 21:05:53.306048 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.305895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:34813 (local address 127.0.0.1:55130)
0927 21:05:53.305954 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.306002 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:34813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:05:53.306180 3829 catalog_manager.cc:4584] TS 3296efda59b74599b85f7724e9afcc7c (127.16.190.193:34813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf9a9f1a618149d2bdc6d95bab498088: Network error: Client connection negotiation failed: client connection to 127.16.190.193:34813: connect: Connection refused (error 111)
I20260927 21:05:53.310169 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4175
I20260927 21:05:53.315006 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 3810
2026-09-27T21:05:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T21:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:53Z Disabled control of system clock
I20260927 21:05:53.343194 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42675
--webserver_interface=127.16.190.254
--webserver_port=0
--builtin_ntp_servers=127.16.190.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:53.424090 4351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:53.424326 4351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:53.424346 4351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:53.425575 4351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:53.425623 4351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:53.425635 4351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:53.425650 4351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:53.425662 4351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:53.426992 4351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:53.427263 4351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:53.427625 4351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:53.432713 4356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:53.432700 4359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.432749 4351 server_base.cc:1034] running on GCE node
W20260927 21:05:53.432722 4357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.433322 4351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:53.433589 4351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:53.434732 4351 hybrid_clock.cc:648] HybridClock initialized: now 1790543153434715 us; error 44 us; skew 500 ppm
I20260927 21:05:53.435994 4351 webserver.cc:460] Webserver started at http://127.16.190.254:43543/ using document root <none> and password file <none>
I20260927 21:05:53.436193 4351 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:53.436234 4351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:53.436324 4351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:53.437448 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "4d2c5ecbf45d455fbc76dc52cf80e009"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "1184269c2181a0f8c6a18b0cc8081847"
server_key_iv: "c45d035dafed38b98b321924b27b0ae3"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.437758 4351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "4d2c5ecbf45d455fbc76dc52cf80e009"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "1184269c2181a0f8c6a18b0cc8081847"
server_key_iv: "c45d035dafed38b98b321924b27b0ae3"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.439106 4351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 21:05:53.439785 4365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.439983 4351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:53.440076 4351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "4d2c5ecbf45d455fbc76dc52cf80e009"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "1184269c2181a0f8c6a18b0cc8081847"
server_key_iv: "c45d035dafed38b98b321924b27b0ae3"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.440136 4351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:53.447890 4351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:53.448175 4351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:53.448282 4351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:53.452142 4351 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:42675
I20260927 21:05:53.452190 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:42675 every 8 connection(s)
I20260927 21:05:53.452600 4351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:53.453626 4419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.455945 4419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: Bootstrap starting.
I20260927 21:05:53.456548 4419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.456931 4419 log.cc:824] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.457715 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4351
I20260927 21:05:53.457808 4419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: No bootstrap required, opened a new log
I20260927 21:05:53.457852 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:53.458036 17147 external_mini_cluster.cc:1428] Setting key 3bae0cb60bab8ad2ec8ba126e222326d
I20260927 21:05:53.459753 4419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } }
I20260927 21:05:53.459939 4419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.459978 4419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d2c5ecbf45d455fbc76dc52cf80e009, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.460134 4419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } }
I20260927 21:05:53.460228 4419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:53.460271 4419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:53.460326 4419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.460999 4419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } }
I20260927 21:05:53.461114 4419 leader_election.cc:302] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d2c5ecbf45d455fbc76dc52cf80e009; no voters:
I20260927 21:05:53.461336 4419 leader_election.cc:288] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:53.461370 4424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:53.461494 4424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [term 1 LEADER]: Becoming Leader. State: Replica: 4d2c5ecbf45d455fbc76dc52cf80e009, State: Running, Role: LEADER
I20260927 21:05:53.461616 4419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:53.461643 4424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } }
I20260927 21:05:53.462001 4424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d2c5ecbf45d455fbc76dc52cf80e009. Latest consensus state: current_term: 1 leader_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } } }
I20260927 21:05:53.462157 4424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:53.462126 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d2c5ecbf45d455fbc76dc52cf80e009" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42675 } } }
I20260927 21:05:53.462435 4432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:53.462456 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:53.463034 4432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:53.465066 4432 catalog_manager.cc:1347] Generated new cluster ID: 2de6e21aa5d140fa89743a12ec618320
I20260927 21:05:53.465123 4432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:53.465795 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.458623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.16.190.254:42675)
0927 21:05:53.458867 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.458883 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.458914 (+ 31us) server_negotiation.cc:407] Connection header received
0927 21:05:53.459736 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.459745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.459816 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.459973 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.460203 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.461951 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.463132 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.463856 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.464009 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.464943 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.464946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.464947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:53.465155 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.465200 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:53.465215 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.465290 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.465398 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.465586 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260927 21:05:53.474311 4432 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:53.475020 4432 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:53.484395 4432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: Generated new TSK 0
I20260927 21:05:53.484615 4432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:53.490823 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.193:0
--local_ip_for_outbound_sockets=127.16.190.193
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--builtin_ntp_servers=127.16.190.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 21:05:53.573150 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:53.573413 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:53.573448 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:53.574623 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:53.574702 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:53.574746 4443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.193
I20260927 21:05:53.576145 4443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.190.193
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.193
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:53.576438 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:53.576762 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:53.583123 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:53.583148 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.583405 4443 server_base.cc:1034] running on GCE node
W20260927 21:05:53.583148 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.583667 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:53.583950 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:53.585124 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790543153585109 us; error 31 us; skew 500 ppm
I20260927 21:05:53.586673 4443 webserver.cc:460] Webserver started at http://127.16.190.193:42691/ using document root <none> and password file <none>
I20260927 21:05:53.586973 4443 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:53.587037 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:53.587173 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:53.588755 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data/instance:
uuid: "f196568f3b4344eab0594b918efd00dd"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "0ab08b782db4963350881b2243d9c9f1"
server_key_iv: "c3494deb31b147af3782f9631035c50c"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.589304 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance:
uuid: "f196568f3b4344eab0594b918efd00dd"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "0ab08b782db4963350881b2243d9c9f1"
server_key_iv: "c3494deb31b147af3782f9631035c50c"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.591045 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:53.591936 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.592171 4443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:53.592257 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal
uuid: "f196568f3b4344eab0594b918efd00dd"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "0ab08b782db4963350881b2243d9c9f1"
server_key_iv: "c3494deb31b147af3782f9631035c50c"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.592327 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:53.604638 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:53.604995 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:53.605106 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:53.605301 4443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:53.605575 4443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:53.605604 4443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.605626 4443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:53.605639 4443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.611513 4443 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.193:36869
I20260927 21:05:53.611560 4571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.193:36869 every 8 connection(s)
I20260927 21:05:53.611907 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/data/info.pb
I20260927 21:05:53.616091 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4443
I20260927 21:05:53.616223 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-0/wal/instance
I20260927 21:05:53.616482 17147 external_mini_cluster.cc:1428] Setting key 209aa152079ebc197aa2310869f3e3db
I20260927 21:05:53.617554 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.612650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:54141 (local address 127.16.190.254:42675)
0927 21:05:53.612712 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.612714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.613027 (+ 313us) server_negotiation.cc:407] Connection header received
0927 21:05:53.613586 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.613588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.613604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.613668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.614355 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.614867 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.615887 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.616084 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.616158 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.617233 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.617234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.617234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:53.617239 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.617255 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:53.617260 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.617310 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.617404 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.617438 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:05:53.617573 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.194:0
--local_ip_for_outbound_sockets=127.16.190.194
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--builtin_ntp_servers=127.16.190.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:05:53.617690 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.612644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42675 (local address 127.16.190.193:54141)
0927 21:05:53.612908 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.612941 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.613473 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.613741 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.613745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.613755 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:53.614256 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.614260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.614968 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.614969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.615775 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.615777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.616178 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.616179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.616363 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.617054 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:53.617054 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:53.617084 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:53.617109 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:53.617316 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:53.617322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:53.617324 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.617404 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.617491 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260927 21:05:53.618149 4572 heartbeater.cc:347] Connected to a master server at 127.16.190.254:42675
I20260927 21:05:53.618252 4572 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:53.618453 4572 heartbeater.cc:511] Master 127.16.190.254:42675 requested a full tablet report, sending...
I20260927 21:05:53.618988 4383 ts_manager.cc:194] Registered new tserver with Master: f196568f3b4344eab0594b918efd00dd (127.16.190.193:36869)
I20260927 21:05:53.619608 4383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.193:54141
W20260927 21:05:53.700160 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:53.700377 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:53.700402 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:53.701638 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:53.701686 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:53.701713 4575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.194
I20260927 21:05:53.703097 4575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.190.194
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.194
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:53.703362 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:53.703670 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:53.708737 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.708833 4575 server_base.cc:1034] running on GCE node
W20260927 21:05:53.708850 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:53.708737 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.709205 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:53.709488 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:53.710681 4575 hybrid_clock.cc:648] HybridClock initialized: now 1790543153710654 us; error 39 us; skew 500 ppm
I20260927 21:05:53.712117 4575 webserver.cc:460] Webserver started at http://127.16.190.194:42175/ using document root <none> and password file <none>
I20260927 21:05:53.712477 4575 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:53.712565 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:53.712740 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:53.713979 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data/instance:
uuid: "8b32301b8d7a4f88bf0f765841123eca"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "7dbe1ebdbd5cf70eb04464190f3782fd"
server_key_iv: "e492840ad49bd221e2ad0cd7fee092df"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.714394 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance:
uuid: "8b32301b8d7a4f88bf0f765841123eca"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "7dbe1ebdbd5cf70eb04464190f3782fd"
server_key_iv: "e492840ad49bd221e2ad0cd7fee092df"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.715865 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:53.716774 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.717005 4575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:53.717120 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal
uuid: "8b32301b8d7a4f88bf0f765841123eca"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "7dbe1ebdbd5cf70eb04464190f3782fd"
server_key_iv: "e492840ad49bd221e2ad0cd7fee092df"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.717219 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:53.731246 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:53.731614 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:53.731791 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:53.732060 4575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:53.732424 4575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:53.732484 4575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.732532 4575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:53.732559 4575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.738950 4575 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.194:37773
I20260927 21:05:53.738989 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.194:37773 every 8 connection(s)
I20260927 21:05:53.739394 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/data/info.pb
I20260927 21:05:53.743005 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4575
I20260927 21:05:53.743155 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal/instance
I20260927 21:05:53.743516 17147 external_mini_cluster.cc:1428] Setting key 579434979776dd249a6e4e33251da8d7
I20260927 21:05:53.744544 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.190.195:0
--local_ip_for_outbound_sockets=127.16.190.195
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--builtin_ntp_servers=127.16.190.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 21:05:53.745400 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.740221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.194:39709 (local address 127.16.190.254:42675)
0927 21:05:53.740294 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.740296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.740537 (+ 241us) server_negotiation.cc:407] Connection header received
0927 21:05:53.741124 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.741125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.741135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.741204 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.741902 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.742465 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.743637 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.743909 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.744001 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.745076 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.745078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.745078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:53.745083 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.745099 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:53.745107 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.745180 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.745282 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.745311 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:05:53.745685 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.740195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42675 (local address 127.16.190.194:39709)
0927 21:05:53.740402 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.740444 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.741008 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.741281 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.741285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.741293 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:53.741785 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.741789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.742553 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.742554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.743536 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.743539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.744011 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.744012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.744164 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.744884 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:53.744885 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:53.744931 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:53.744957 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:53.745183 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:53.745189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:53.745193 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.745281 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.745395 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260927 21:05:53.746134 4705 heartbeater.cc:347] Connected to a master server at 127.16.190.254:42675
I20260927 21:05:53.746222 4705 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:53.746448 4705 heartbeater.cc:511] Master 127.16.190.254:42675 requested a full tablet report, sending...
I20260927 21:05:53.746843 4383 ts_manager.cc:194] Registered new tserver with Master: 8b32301b8d7a4f88bf0f765841123eca (127.16.190.194:37773)
I20260927 21:05:53.747231 4383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.194:39709
W20260927 21:05:53.828102 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:53.828331 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:53.828354 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:53.829722 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:53.829769 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:53.829797 4708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.190.195
I20260927 21:05:53.831233 4708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.190.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.190.195
--webserver_port=0
--tserver_master_addrs=127.16.190.254:42675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.190.195
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:53.831447 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:53.831697 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:53.836916 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:53.836966 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.837085 4708 server_base.cc:1034] running on GCE node
W20260927 21:05:53.836933 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:53.837445 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:53.837750 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:53.838958 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790543153838948 us; error 35 us; skew 500 ppm
I20260927 21:05:53.840346 4708 webserver.cc:460] Webserver started at http://127.16.190.195:35017/ using document root <none> and password file <none>
I20260927 21:05:53.840612 4708 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:53.840687 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:53.840885 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:53.842017 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data/instance:
uuid: "f00a1f5bfe74405998a47549169f1a73"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "ac6078e941d4c5ec2fe1f2b46b3cc8bd"
server_key_iv: "c74a090caf99a8f59dc0e9640f4de9fa"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.842418 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance:
uuid: "f00a1f5bfe74405998a47549169f1a73"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "ac6078e941d4c5ec2fe1f2b46b3cc8bd"
server_key_iv: "c74a090caf99a8f59dc0e9640f4de9fa"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.843842 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:53.844558 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.844732 4708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 21:05:53.844856 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal
uuid: "f00a1f5bfe74405998a47549169f1a73"
format_stamp: "Formatted at 2026-09-27 21:05:53 on dist-test-slave-6c7s"
server_key: "ac6078e941d4c5ec2fe1f2b46b3cc8bd"
server_key_iv: "c74a090caf99a8f59dc0e9640f4de9fa"
server_key_version: "encryptionkey@0"
I20260927 21:05:53.844935 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:53.854840 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:53.855175 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:53.855333 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:53.855567 4708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 21:05:53.855909 4708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 21:05:53.855966 4708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.856006 4708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 21:05:53.856035 4708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:53.862151 4708 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.195:34221
I20260927 21:05:53.862191 4836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.195:34221 every 8 connection(s)
I20260927 21:05:53.862591 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/data/info.pb
I20260927 21:05:53.867877 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.863361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.195:37077 (local address 127.16.190.254:42675)
0927 21:05:53.863406 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.863408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.863682 (+ 274us) server_negotiation.cc:407] Connection header received
0927 21:05:53.864155 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.864156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.864167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.864221 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.864933 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.865490 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.866475 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.866664 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.866723 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.867605 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.867607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.867607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:53.867612 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.867627 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:53.867632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.867699 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.867782 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.867810 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:05:53.868099 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.863380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42675 (local address 127.16.190.195:37077)
0927 21:05:53.863551 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.863588 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.864070 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.864274 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.864277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.864285 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:05:53.864775 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.864780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.865583 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.865583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.866385 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.866387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.866744 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.866745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.866859 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.867444 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 21:05:53.867445 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:05:53.867475 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 21:05:53.867499 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:05:53.867706 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:05:53.867710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:05:53.867712 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.867765 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.867844 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260927 21:05:53.868705 4837 heartbeater.cc:347] Connected to a master server at 127.16.190.254:42675
I20260927 21:05:53.868813 4837 heartbeater.cc:464] Registering TS with master...
I20260927 21:05:53.869022 4837 heartbeater.cc:511] Master 127.16.190.254:42675 requested a full tablet report, sending...
I20260927 21:05:53.869534 4383 ts_manager.cc:194] Registered new tserver with Master: f00a1f5bfe74405998a47549169f1a73 (127.16.190.195:34221)
I20260927 21:05:53.869918 4383 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.190.195:37077
I20260927 21:05:53.870607 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4708
I20260927 21:05:53.870745 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal/instance
I20260927 21:05:53.870986 17147 external_mini_cluster.cc:1428] Setting key 864a52c36bfeefc605cbd89e4116e297
I20260927 21:05:53.871845 17147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 21:05:53.876348 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.873369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.16.190.254:42675)
0927 21:05:53.873454 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.873456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.873565 (+ 109us) server_negotiation.cc:407] Connection header received
0927 21:05:53.873590 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.873591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.873604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.873668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:53.873830 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.874327 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.875203 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.875417 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.875479 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.876146 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:53.876147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:53.876147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:53.876152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:53.876164 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:53.876168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:53.876212 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.876246 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.876268 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 21:05:53.878273 4383 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 21:05:53.880100 4383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 21:05:53.887368 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.882827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.16.190.194:37773)
0927 21:05:53.883271 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.883275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.883283 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:05:53.883336 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.883338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.883356 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.883656 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.883717 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.884869 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.886207 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.886952 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.887038 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.887129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.887262 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.887299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":101,"threads_started":1}
I20260927 21:05:53.887388 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.882698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37773 (local address 127.0.0.1:49386)
0927 21:05:53.882885 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.882938 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.883008 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.883474 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.883477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.883480 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.883576 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.883578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.884946 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.884947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.886114 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.886116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.887059 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.887060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.887144 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.887159 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.887251 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.887306 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":8,"thread_start_us":126,"threads_started":1}
I20260927 21:05:53.888199 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.882942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:36869 (local address 127.0.0.1:45198)
0927 21:05:53.883197 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.883206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.883262 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.883549 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.883550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.883552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.883581 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.883582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.886221 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.886222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.886982 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.886984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.888047 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.888048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.888096 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.888099 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.888128 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.888155 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":55,"threads_started":1}
I20260927 21:05:53.888286 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.883066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.16.190.193:36869)
0927 21:05:53.883244 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.883250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.883262 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:53.883397 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.883399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.883413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.883485 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.883656 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.884854 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.887113 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.887957 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.888035 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.888124 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.888151 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.888191 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260927 21:05:53.888255 4638 tablet_service.cc:1467] Processing CreateTablet for tablet c5e39324a3b444d9baf51bc21d832be9 (DEFAULT_TABLE table=test-table [id=cd3fcd983f44421d852caf72df361e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.888693 4638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e39324a3b444d9baf51bc21d832be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.888964 4505 tablet_service.cc:1467] Processing CreateTablet for tablet c5e39324a3b444d9baf51bc21d832be9 (DEFAULT_TABLE table=test-table [id=cd3fcd983f44421d852caf72df361e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.889217 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.883075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.16.190.195:34221)
0927 21:05:53.883280 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.883285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.883297 (+ 12us) server_negotiation.cc:407] Connection header received
0927 21:05:53.883340 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.883342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.883358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.883666 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.883739 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.884481 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.886208 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.886967 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.887038 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.887129 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.889064 (+ 1935us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.889124 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260927 21:05:53.889218 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.882700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34221 (local address 127.0.0.1:45702)
0927 21:05:53.883029 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.883049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.883106 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.883477 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.883478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.883480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.883575 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.883577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.884579 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.884580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.886104 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.886107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.888895 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.888896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.888962 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.888966 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.889029 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.889067 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260927 21:05:53.889336 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e39324a3b444d9baf51bc21d832be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.890245 4770 tablet_service.cc:1467] Processing CreateTablet for tablet c5e39324a3b444d9baf51bc21d832be9 (DEFAULT_TABLE table=test-table [id=cd3fcd983f44421d852caf72df361e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 21:05:53.890620 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e39324a3b444d9baf51bc21d832be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:53.892066 4851 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: Bootstrap starting.
I20260927 21:05:53.892490 4852 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Bootstrap starting.
I20260927 21:05:53.892766 4851 tablet_bootstrap.cc:654] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.893143 4851 log.cc:824] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.893204 4852 tablet_bootstrap.cc:654] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.893519 4852 log.cc:824] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.893900 4851 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: No bootstrap required, opened a new log
I20260927 21:05:53.893988 4851 ts_tablet_manager.cc:1397] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 21:05:53.894304 4852 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: No bootstrap required, opened a new log
I20260927 21:05:53.894383 4852 ts_tablet_manager.cc:1397] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 21:05:53.894667 4855 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Bootstrap starting.
I20260927 21:05:53.895398 4855 tablet_bootstrap.cc:654] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:53.895785 4855 log.cc:824] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:53.896670 4855 tablet_bootstrap.cc:492] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: No bootstrap required, opened a new log
I20260927 21:05:53.896788 4855 ts_tablet_manager.cc:1397] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 21:05:53.896708 4851 raft_consensus.cc:348] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.896932 4851 raft_consensus.cc:374] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.896981 4851 raft_consensus.cc:729] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f196568f3b4344eab0594b918efd00dd, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.897117 4851 consensus_queue.cc:260] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.897416 4851 ts_tablet_manager.cc:1428] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 21:05:53.897554 4572 heartbeater.cc:503] Master 127.16.190.254:42675 was elected leader, sending a full tablet report...
I20260927 21:05:53.897521 4852 raft_consensus.cc:348] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.897706 4852 raft_consensus.cc:374] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.897753 4852 raft_consensus.cc:729] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b32301b8d7a4f88bf0f765841123eca, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.897884 4852 consensus_queue.cc:260] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.898182 4852 ts_tablet_manager.cc:1428] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260927 21:05:53.898238 4705 heartbeater.cc:503] Master 127.16.190.254:42675 was elected leader, sending a full tablet report...
I20260927 21:05:53.899466 4855 raft_consensus.cc:348] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.899590 4855 raft_consensus.cc:374] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:53.899610 4855 raft_consensus.cc:729] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f00a1f5bfe74405998a47549169f1a73, State: Initialized, Role: FOLLOWER
I20260927 21:05:53.899724 4855 consensus_queue.cc:260] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.899971 4855 ts_tablet_manager.cc:1428] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 21:05:53.900035 4837 heartbeater.cc:503] Master 127.16.190.254:42675 was elected leader, sending a full tablet report...
I20260927 21:05:53.965432 4857 raft_consensus.cc:482] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 21:05:53.965620 4857 raft_consensus.cc:504] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.966017 4857 leader_election.cc:288] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f00a1f5bfe74405998a47549169f1a73 (127.16.190.195:34221), 8b32301b8d7a4f88bf0f765841123eca (127.16.190.194:37773)
I20260927 21:05:53.969079 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.966294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:35203 (local address 127.16.190.194:37773)
0927 21:05:53.966366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.966368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.966611 (+ 243us) server_negotiation.cc:407] Connection header received
0927 21:05:53.966687 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.966689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.966701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.966770 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.966923 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.967384 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968306 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.968668 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968771 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.968778 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.968967 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.969008 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 21:05:53.969125 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.966210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.194:37773 (local address 127.16.190.193:35203)
0927 21:05:53.966534 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.966543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.966622 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.966772 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.966773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.966774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.966842 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.966846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.967468 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.967469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.968213 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.968215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.968771 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.968881 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.968885 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.968961 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.969001 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":169,"thread_start_us":107,"threads_started":1}
I20260927 21:05:53.969187 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.966331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.190.193:57351 (local address 127.16.190.195:34221)
0927 21:05:53.966378 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.966380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.966399 (+ 19us) server_negotiation.cc:407] Connection header received
0927 21:05:53.966428 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.966429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.966437 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.966532 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 21:05:53.966703 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.967243 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968312 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.968702 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968846 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.968855 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.969016 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.969077 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:05:53.969406 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.966203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.195:34221 (local address 127.16.190.193:57351)
0927 21:05:53.966249 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.966262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:05:53.966310 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:05:53.966537 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:05:53.966539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:05:53.966540 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 21:05:53.966620 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.966623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.967342 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.967343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.968187 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:05:53.968189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.968843 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.968844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:05:53.968926 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.968929 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 21:05:53.969009 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:05:53.969056 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":27}
I20260927 21:05:53.969547 4658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e39324a3b444d9baf51bc21d832be9" candidate_uuid: "f196568f3b4344eab0594b918efd00dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b32301b8d7a4f88bf0f765841123eca" is_pre_election: true
I20260927 21:05:53.969704 4658 raft_consensus.cc:2454] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f196568f3b4344eab0594b918efd00dd in term 0.
I20260927 21:05:53.969787 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e39324a3b444d9baf51bc21d832be9" candidate_uuid: "f196568f3b4344eab0594b918efd00dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00a1f5bfe74405998a47549169f1a73" is_pre_election: true
I20260927 21:05:53.969947 4790 raft_consensus.cc:2454] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f196568f3b4344eab0594b918efd00dd in term 0.
I20260927 21:05:53.969981 4460 leader_election.cc:302] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b32301b8d7a4f88bf0f765841123eca, f196568f3b4344eab0594b918efd00dd; no voters:
I20260927 21:05:53.970131 4857 raft_consensus.cc:2781] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 21:05:53.970188 4857 raft_consensus.cc:482] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 21:05:53.970219 4857 raft_consensus.cc:3037] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.970968 4857 raft_consensus.cc:504] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.971132 4857 leader_election.cc:288] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [CANDIDATE]: Term 1 election: Requested vote from peers f00a1f5bfe74405998a47549169f1a73 (127.16.190.195:34221), 8b32301b8d7a4f88bf0f765841123eca (127.16.190.194:37773)
I20260927 21:05:53.971308 4658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e39324a3b444d9baf51bc21d832be9" candidate_uuid: "f196568f3b4344eab0594b918efd00dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b32301b8d7a4f88bf0f765841123eca"
I20260927 21:05:53.971310 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e39324a3b444d9baf51bc21d832be9" candidate_uuid: "f196568f3b4344eab0594b918efd00dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f00a1f5bfe74405998a47549169f1a73"
I20260927 21:05:53.971379 4658 raft_consensus.cc:3037] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.971405 4790 raft_consensus.cc:3037] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:53.972052 4658 raft_consensus.cc:2454] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f196568f3b4344eab0594b918efd00dd in term 1.
I20260927 21:05:53.972126 4790 raft_consensus.cc:2454] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f196568f3b4344eab0594b918efd00dd in term 1.
I20260927 21:05:53.972317 4458 leader_election.cc:302] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f00a1f5bfe74405998a47549169f1a73, f196568f3b4344eab0594b918efd00dd; no voters:
I20260927 21:05:53.972415 4857 raft_consensus.cc:2781] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:53.972596 4857 raft_consensus.cc:686] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [term 1 LEADER]: Becoming Leader. State: Replica: f196568f3b4344eab0594b918efd00dd, State: Running, Role: LEADER
I20260927 21:05:53.972704 4857 consensus_queue.cc:237] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } }
I20260927 21:05:53.973595 4383 catalog_manager.cc:5495] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd reported cstate change: term changed from 0 to 1, leader changed from <none> to f196568f3b4344eab0594b918efd00dd (127.16.190.193). New cstate: current_term: 1 leader_uuid: "f196568f3b4344eab0594b918efd00dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f196568f3b4344eab0594b918efd00dd" member_type: VOTER last_known_addr { host: "127.16.190.193" port: 36869 } health_report { overall_health: HEALTHY } } }
I20260927 21:05:53.982177 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.979617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.16.190.193:36869)
0927 21:05:53.979668 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:53.979670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:53.979677 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:05:53.979775 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:53.979776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:53.979792 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:53.979849 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:05:53.979962 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.980408 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.981323 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:53.981508 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:53.981565 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:53.981736 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:05:53.981830 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:05:53.981898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:53.981940 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 21:05:53.981958 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:05:53.984004 4790 raft_consensus.cc:1261] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 1 FOLLOWER]: Refusing update from remote peer f196568f3b4344eab0594b918efd00dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:53.984022 4658 raft_consensus.cc:1261] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 1 FOLLOWER]: Refusing update from remote peer f196568f3b4344eab0594b918efd00dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:05:53.984413 4857 consensus_queue.cc:1035] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f00a1f5bfe74405998a47549169f1a73" member_type: VOTER last_known_addr { host: "127.16.190.195" port: 34221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:53.984553 4861 consensus_queue.cc:1035] T c5e39324a3b444d9baf51bc21d832be9 P f196568f3b4344eab0594b918efd00dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b32301b8d7a4f88bf0f765841123eca" member_type: VOTER last_known_addr { host: "127.16.190.194" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:05:53.987039 4864 mvcc.cc:205] Tried to move back new op lower bound from 7334064758715641856 to 7334064758672953344. Current Snapshot: MvccSnapshot[applied={T|T < 7334064758715641856}]
I20260927 21:05:53.987154 4865 mvcc.cc:205] Tried to move back new op lower bound from 7334064758715641856 to 7334064758672953344. Current Snapshot: MvccSnapshot[applied={T|T < 7334064758715641856}]
I20260927 21:05:53.987349 4866 mvcc.cc:205] Tried to move back new op lower bound from 7334064758715641856 to 7334064758672953344. Current Snapshot: MvccSnapshot[applied={T|T < 7334064758715641856}]
I20260927 21:05:53.989140 4382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:53.989231 4382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 21:05:53.990072 4382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d2c5ecbf45d455fbc76dc52cf80e009: Sending DeleteTablet for 3 replicas of tablet c5e39324a3b444d9baf51bc21d832be9
I20260927 21:05:53.990491 4770 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e39324a3b444d9baf51bc21d832be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:53 UTC) from {username='slave'} at 127.0.0.1:45702
I20260927 21:05:53.990489 4638 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e39324a3b444d9baf51bc21d832be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 21:05:53 UTC) from {username='slave'} at 127.0.0.1:49386
I20260927 21:05:53.990595 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4443
I20260927 21:05:53.991279 4874 tablet_replica.cc:330] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: stopping tablet replica
I20260927 21:05:53.991361 4874 raft_consensus.cc:2229] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:53.991436 4874 raft_consensus.cc:2258] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:53.991601 4875 tablet_replica.cc:330] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: stopping tablet replica
I20260927 21:05:53.991690 4875 raft_consensus.cc:2229] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 21:05:53.991786 4875 raft_consensus.cc:2258] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 21:05:53.991897 4874 ts_tablet_manager.cc:1905] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:53.992213 4875 ts_tablet_manager.cc:1905] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 21:05:53.994050 4874 ts_tablet_manager.cc:1918] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:53.994048 4875 ts_tablet_manager.cc:1918] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 21:05:53.994127 4875 log.cc:1196] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-1/wal/wals/c5e39324a3b444d9baf51bc21d832be9
I20260927 21:05:53.994138 4874 log.cc:1196] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Deleting WAL directory at /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790542906842326-17147-0/minicluster-data/ts-2/wal/wals/c5e39324a3b444d9baf51bc21d832be9
I20260927 21:05:53.994418 4874 ts_tablet_manager.cc:1939] T c5e39324a3b444d9baf51bc21d832be9 P f00a1f5bfe74405998a47549169f1a73: Deleting consensus metadata
I20260927 21:05:53.994428 4875 ts_tablet_manager.cc:1939] T c5e39324a3b444d9baf51bc21d832be9 P 8b32301b8d7a4f88bf0f765841123eca: Deleting consensus metadata
I20260927 21:05:53.995559 4366 catalog_manager.cc:4841] TS f00a1f5bfe74405998a47549169f1a73 (127.16.190.195:34221): tablet c5e39324a3b444d9baf51bc21d832be9 (table test-table [id=cd3fcd983f44421d852caf72df361e5e]) successfully deleted
I20260927 21:05:53.995563 4368 catalog_manager.cc:4841] TS 8b32301b8d7a4f88bf0f765841123eca (127.16.190.194:37773): tablet c5e39324a3b444d9baf51bc21d832be9 (table test-table [id=cd3fcd983f44421d852caf72df361e5e]) successfully deleted
W20260927 21:05:53.998109 4368 connection.cc:535] client connection to 127.16.190.193:36869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 21:05:53.998202 4368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 21:05:53.998342 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4575
I20260927 21:05:53.999729 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:53.999541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.193:36869 (local address 127.0.0.1:45222)
0927 21:05:53.999625 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 21:05:53.999675 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.193:36869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 21:05:53.999895 4368 catalog_manager.cc:4584] TS f196568f3b4344eab0594b918efd00dd (127.16.190.193:36869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5e39324a3b444d9baf51bc21d832be9: Network error: Client connection negotiation failed: client connection to 127.16.190.193:36869: connect: Connection refused (error 111)
I20260927 21:05:54.003538 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4708
I20260927 21:05:54.008194 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4351
2026-09-27T21:05:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (695 ms)
[----------] 6 tests from EncryptionPolicyTest (8308 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:54 dist-test-slave-6c7s krb5kdc[4879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:54 dist-test-slave-6c7s krb5kdc[4879](info): set up 2 sockets
Sep 27 21:05:54 dist-test-slave-6c7s krb5kdc[4879](info): commencing operation
krb5kdc: starting...
W20260927 21:05:55.053798 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:05:55.180725 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:55.263525 4892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:55.263747 4892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:55.263772 4892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:55.264916 4892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:55.264956 4892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:55.264971 4892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:55.264983 4892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:55.264995 4892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:55.266541 4892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:55.266839 4892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:55.267069 4892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:55.272039 4897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:55.272089 4892 server_base.cc:1034] running on GCE node
W20260927 21:05:55.272045 4898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:55.272045 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:55.272570 4892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:55.272953 4892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:55.274133 4892 hybrid_clock.cc:648] HybridClock initialized: now 1790543155274127 us; error 40 us; skew 500 ppm
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:55.276646 4892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:05:55.277166 4892 webserver.cc:460] Webserver started at http://127.0.0.1:39421/ using document root <none> and password file <none>
I20260927 21:05:55.277422 4892 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:55.277491 4892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:55.277652 4892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:55.278723 4892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "5cdd85fa72bd4e5b9e041034c86ffb41"
format_stamp: "Formatted at 2026-09-27 21:05:55 on dist-test-slave-6c7s"
server_key: "aea1815c69f0668eff01dcb042a6e745"
server_key_iv: "e4bfdadd1107d3321c0ab3f07fe40e5f"
server_key_version: "encryptionkey@0"
I20260927 21:05:55.279084 4892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "5cdd85fa72bd4e5b9e041034c86ffb41"
format_stamp: "Formatted at 2026-09-27 21:05:55 on dist-test-slave-6c7s"
server_key: "aea1815c69f0668eff01dcb042a6e745"
server_key_iv: "e4bfdadd1107d3321c0ab3f07fe40e5f"
server_key_version: "encryptionkey@0"
I20260927 21:05:55.280491 4892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:05:55.281323 4907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:55.281504 4892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:05:55.281603 4892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "5cdd85fa72bd4e5b9e041034c86ffb41"
format_stamp: "Formatted at 2026-09-27 21:05:55 on dist-test-slave-6c7s"
server_key: "aea1815c69f0668eff01dcb042a6e745"
server_key_iv: "e4bfdadd1107d3321c0ab3f07fe40e5f"
server_key_version: "encryptionkey@0"
I20260927 21:05:55.281683 4892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:55.296118 4892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:55.298739 4892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:55.298954 4892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:55.303010 4892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40489
I20260927 21:05:55.303093 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40489 every 8 connection(s)
I20260927 21:05:55.303460 4892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:55.304392 4960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:55.305923 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 4892
I20260927 21:05:55.306039 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:55.306262 17147 external_mini_cluster.cc:1428] Setting key 848bab7643da4ca4d52bf69a688ccd6f
I20260927 21:05:55.307305 4960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: Bootstrap starting.
I20260927 21:05:55.307928 4960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:55.308255 4960 log.cc:824] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:55.309083 4960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: No bootstrap required, opened a new log
I20260927 21:05:55.311317 4960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260927 21:05:55.311506 4960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:55.311555 4960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdd85fa72bd4e5b9e041034c86ffb41, State: Initialized, Role: FOLLOWER
I20260927 21:05:55.311707 4960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260927 21:05:55.311785 4960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:55.311829 4960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:55.311877 4960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:55.312423 4960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260927 21:05:55.312544 4960 leader_election.cc:302] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cdd85fa72bd4e5b9e041034c86ffb41; no voters:
I20260927 21:05:55.312755 4960 leader_election.cc:288] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:55.312770 4965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:55.312913 4965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [term 1 LEADER]: Becoming Leader. State: Replica: 5cdd85fa72bd4e5b9e041034c86ffb41, State: Running, Role: LEADER
I20260927 21:05:55.312995 4965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } }
I20260927 21:05:55.313050 4960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:55.313330 4966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260927 21:05:55.313427 4966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [sys.catalog]: This master's current role is: LEADER
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:05:55.313594 4967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cdd85fa72bd4e5b9e041034c86ffb41. Latest consensus state: current_term: 1 leader_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdd85fa72bd4e5b9e041034c86ffb41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40489 } } }
I20260927 21:05:55.313655 4967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41 [sys.catalog]: This master's current role is: LEADER
W20260927 21:05:55.314760 4981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:05:55.314815 4981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:05:55.314873 4972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:55.315019 4972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:55.316571 4972 catalog_manager.cc:1347] Generated new cluster ID: 6527d6b40c3e481082c689bb9f14583b
I20260927 21:05:55.316627 4972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:55.317613 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:55.306903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56093 (local address 127.0.0.1:40489)
0927 21:05:55.307084 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:55.307092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:55.307117 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:05:55.307613 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:55.307620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:55.307671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:55.307794 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:55.308627 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:55.309686 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:55.310779 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:55.311178 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:55.311258 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:55.314060 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:55.314064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:55.314064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:55.314084 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:55.316014 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:55.316419 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:55.316422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:55.316422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:55.316537 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:55.316748 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:55.316750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:55.316750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:55.317041 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:55.317210 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:55.317341 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 21:05:55.317451 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260927 21:05:55.321942 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:05:55.335788 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:55.329154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36623 (local address 127.0.0.1:40489)
0927 21:05:55.329200 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:55.329202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:55.329393 (+ 191us) server_negotiation.cc:407] Connection header received
0927 21:05:55.329455 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:55.329456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:55.329466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:55.329510 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:55.330098 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:55.330634 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:55.331450 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:55.331671 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:55.331757 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:55.333160 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:55.333161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:55.333162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:55.333170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:55.334596 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:55.334978 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:55.334979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:55.334980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:55.335043 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:55.335264 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:55.335265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:55.335265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:55.335383 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:55.335495 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:55.335615 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 21:05:55.335682 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:05:55.338225 4972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:55.338919 4972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:55.350426 4972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cdd85fa72bd4e5b9e041034c86ffb41: Generated new TSK 0
I20260927 21:05:55.350661 4972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:55.355296 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 4892
W20260927 21:05:55.360565 4877 connection.cc:535] client connection to 127.0.0.1:40489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40489): Connection reset by peer (error 104)
2026-09-27T21:05:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4992](info): set up 2 sockets
Sep 27 21:05:55 dist-test-slave-6c7s krb5kdc[4992](info): commencing operation
krb5kdc: starting...
W20260927 21:05:56.397807 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:56 dist-test-slave-6c7s krb5kdc[4992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:05:56.530145 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:56.615528 5005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:56.615775 5005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:56.615803 5005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:56.617048 5005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:56.617096 5005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:56.617111 5005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:56.617125 5005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:56.617136 5005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:56.618680 5005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:56.618927 5005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:56.619112 5005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:56.624575 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:56.624575 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:56.624568 5013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:56.624950 5005 server_base.cc:1034] running on GCE node
I20260927 21:05:56.625216 5005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:56.625560 5005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:56.626775 5005 hybrid_clock.cc:648] HybridClock initialized: now 1790543156626715 us; error 86 us; skew 500 ppm
Sep 27 21:05:56 dist-test-slave-6c7s krb5kdc[4992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:56.633867 5005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:05:56.634410 5005 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260927 21:05:56.634668 5005 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:56.634750 5005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:56.634927 5005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:56.636116 5005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "2214f4bdff934eed87ad2f2113de9136"
format_stamp: "Formatted at 2026-09-27 21:05:56 on dist-test-slave-6c7s"
server_key: "d4bcc21c5ed82f4a858f6316650d77f5"
server_key_iv: "e18de24425945d3b722995aef8c95cb7"
server_key_version: "encryptionkey@0"
I20260927 21:05:56.636524 5005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "2214f4bdff934eed87ad2f2113de9136"
format_stamp: "Formatted at 2026-09-27 21:05:56 on dist-test-slave-6c7s"
server_key: "d4bcc21c5ed82f4a858f6316650d77f5"
server_key_iv: "e18de24425945d3b722995aef8c95cb7"
server_key_version: "encryptionkey@0"
I20260927 21:05:56.638535 5005 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 21:05:56.639564 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:56.639909 5005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:56.640062 5005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "2214f4bdff934eed87ad2f2113de9136"
format_stamp: "Formatted at 2026-09-27 21:05:56 on dist-test-slave-6c7s"
server_key: "d4bcc21c5ed82f4a858f6316650d77f5"
server_key_iv: "e18de24425945d3b722995aef8c95cb7"
server_key_version: "encryptionkey@0"
I20260927 21:05:56.640162 5005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:56.651659 5005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:56.654188 5005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:56.654383 5005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:56.659207 5005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33485
I20260927 21:05:56.659243 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33485 every 8 connection(s)
I20260927 21:05:56.659657 5005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:56.660665 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:56.663149 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136: Bootstrap starting.
I20260927 21:05:56.663924 5073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:56.664300 5073 log.cc:824] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:56.665331 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136: No bootstrap required, opened a new log
I20260927 21:05:56.665673 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5005
I20260927 21:05:56.665808 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:56.665998 17147 external_mini_cluster.cc:1428] Setting key fe96e83674f20560afa5493c4f275ddf
I20260927 21:05:56.668599 5073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } }
I20260927 21:05:56.668891 5073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:56.668946 5073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2214f4bdff934eed87ad2f2113de9136, State: Initialized, Role: FOLLOWER
I20260927 21:05:56.669132 5073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } }
I20260927 21:05:56.669282 5073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:56.669340 5073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:56.669399 5073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:56.670097 5073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } }
I20260927 21:05:56.670238 5073 leader_election.cc:302] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2214f4bdff934eed87ad2f2113de9136; no voters:
I20260927 21:05:56.670445 5073 leader_election.cc:288] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:56.670468 5078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:56.670595 5078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [term 1 LEADER]: Becoming Leader. State: Replica: 2214f4bdff934eed87ad2f2113de9136, State: Running, Role: LEADER
I20260927 21:05:56.670702 5078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } }
I20260927 21:05:56.670754 5073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:56.671072 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2214f4bdff934eed87ad2f2113de9136. Latest consensus state: current_term: 1 leader_uuid: "2214f4bdff934eed87ad2f2113de9136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } } }
I20260927 21:05:56.671178 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:56.671066 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2214f4bdff934eed87ad2f2113de9136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2214f4bdff934eed87ad2f2113de9136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33485 } } }
I20260927 21:05:56.671339 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:56.671446 5084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:56.672056 5084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:56.673754 5095 mvcc.cc:205] Tried to move back new op lower bound from 7334064769733103616 to 7334064769723469824. Current Snapshot: MvccSnapshot[applied={T|T < 7334064769733103616}]
I20260927 21:05:56.673995 5084 catalog_manager.cc:1347] Generated new cluster ID: d3707637e69c4d089354c3adb658ce8e
I20260927 21:05:56.674042 5084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:56.674489 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:56.666593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56491 (local address 127.0.0.1:33485)
0927 21:05:56.666995 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:56.667007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:56.667048 (+ 41us) server_negotiation.cc:407] Connection header received
0927 21:05:56.668521 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:56.668533 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:56.668652 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:56.668895 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:56.669108 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:56.670548 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:56.671643 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:56.672313 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:56.672788 (+ 475us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:56.673431 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:56.673435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:56.673435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:56.673647 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:56.673687 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:56.673703 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:56.674196 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:56.674243 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 21:05:56.674275 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":92,"threads_started":1}
I20260927 21:05:56.675736 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:05:56.681783 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:56.677812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38515 (local address 127.0.0.1:33485)
0927 21:05:56.677864 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:56.677865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:56.677875 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:05:56.677901 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:56.677902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:56.677912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:56.677992 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:56.678172 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:56.679061 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:56.680197 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:56.680562 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:56.680645 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:56.681493 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:56.681494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:56.681495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:56.681501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:56.681519 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:56.681525 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:56.681590 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:56.681673 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 21:05:56.681688 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:05:56.695339 5084 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:56.695986 5084 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:56.702579 5084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2214f4bdff934eed87ad2f2113de9136: Generated new TSK 0
I20260927 21:05:56.702843 5084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:56.712342 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5005
W20260927 21:05:56.716970 4990 connection.cc:535] client connection to 127.0.0.1:33485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33485): Connection reset by peer (error 104)
2026-09-27T21:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:56 dist-test-slave-6c7s krb5kdc[5104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:56 dist-test-slave-6c7s krb5kdc[5104](info): set up 2 sockets
Sep 27 21:05:56 dist-test-slave-6c7s krb5kdc[5104](info): commencing operation
krb5kdc: starting...
W20260927 21:05:57.749694 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:57 dist-test-slave-6c7s krb5kdc[5104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:05:57.881868 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:57.963732 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:57.963958 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:57.963981 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:57.965186 5118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:57.965234 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:57.965248 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:57.965261 5118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:57.965274 5118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:57.966873 5118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:57.967144 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:57.967377 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:57.972611 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:57.972612 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:57.972612 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:57.972654 5118 server_base.cc:1034] running on GCE node
I20260927 21:05:57.973186 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:57.973455 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:57.974593 5118 hybrid_clock.cc:648] HybridClock initialized: now 1790543157974568 us; error 42 us; skew 500 ppm
Sep 27 21:05:57 dist-test-slave-6c7s krb5kdc[5104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:57.977160 5118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:05:57.977571 5118 webserver.cc:460] Webserver started at http://127.0.0.1:41137/ using document root <none> and password file <none>
I20260927 21:05:57.977816 5118 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:57.977864 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:57.977955 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:57.979014 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "291c2e02caca419694ba848ebc27d7c5"
format_stamp: "Formatted at 2026-09-27 21:05:57 on dist-test-slave-6c7s"
server_key: "9baedde65ea861ea8607c7e91ba78ee6"
server_key_iv: "09cded4b898aaceb1ab944e4206772a2"
server_key_version: "encryptionkey@0"
I20260927 21:05:57.979357 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "291c2e02caca419694ba848ebc27d7c5"
format_stamp: "Formatted at 2026-09-27 21:05:57 on dist-test-slave-6c7s"
server_key: "9baedde65ea861ea8607c7e91ba78ee6"
server_key_iv: "09cded4b898aaceb1ab944e4206772a2"
server_key_version: "encryptionkey@0"
I20260927 21:05:57.980773 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:05:57.981673 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:57.981899 5118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:57.981988 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "291c2e02caca419694ba848ebc27d7c5"
format_stamp: "Formatted at 2026-09-27 21:05:57 on dist-test-slave-6c7s"
server_key: "9baedde65ea861ea8607c7e91ba78ee6"
server_key_iv: "09cded4b898aaceb1ab944e4206772a2"
server_key_version: "encryptionkey@0"
I20260927 21:05:57.982048 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:57.985888 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:57.988292 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:57.988413 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:57.992599 5118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44227
I20260927 21:05:57.992676 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44227 every 8 connection(s)
I20260927 21:05:57.993077 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:57.994154 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:57.996743 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5118
I20260927 21:05:57.996958 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:57.996945 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5: Bootstrap starting.
I20260927 21:05:57.997197 17147 external_mini_cluster.cc:1428] Setting key b184f7cc74824bc0ac2dedc3318da4cc
I20260927 21:05:57.997694 5186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:57.998087 5186 log.cc:824] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:57.998893 5186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5: No bootstrap required, opened a new log
I20260927 21:05:58.000515 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:57.997864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57843 (local address 127.0.0.1:44227)
0927 21:05:57.998108 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:57.998122 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:57.998173 (+ 51us) server_negotiation.cc:407] Connection header received
0927 21:05:57.998780 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:57.998789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:57.998821 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:57.998984 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:57.999820 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:57.999823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:57.999824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:57.999829 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:57.999881 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:57.999898 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:58.000004 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:58.000090 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 21:05:58.000139 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
I20260927 21:05:58.001780 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:05:58.001657 5186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } }
I20260927 21:05:58.001861 5186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:58.001894 5186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291c2e02caca419694ba848ebc27d7c5, State: Initialized, Role: FOLLOWER
I20260927 21:05:58.002032 5186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } }
I20260927 21:05:58.002116 5186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:58.002158 5186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:58.002213 5186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:58.003110 5186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } }
I20260927 21:05:58.003239 5186 leader_election.cc:302] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 291c2e02caca419694ba848ebc27d7c5; no voters:
I20260927 21:05:58.003402 5186 leader_election.cc:288] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:58.003443 5195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:58.003551 5195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [term 1 LEADER]: Becoming Leader. State: Replica: 291c2e02caca419694ba848ebc27d7c5, State: Running, Role: LEADER
I20260927 21:05:58.003643 5195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } }
I20260927 21:05:58.003670 5186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:58.003986 5195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "291c2e02caca419694ba848ebc27d7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } } }
I20260927 21:05:58.004071 5195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:58.004213 5195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291c2e02caca419694ba848ebc27d7c5. Latest consensus state: current_term: 1 leader_uuid: "291c2e02caca419694ba848ebc27d7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291c2e02caca419694ba848ebc27d7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44227 } } }
I20260927 21:05:58.004279 5202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:58.004280 5195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:58.004931 5202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:58.005167 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:58.003515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52849 (local address 127.0.0.1:44227)
0927 21:05:58.003618 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:58.003620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:58.003805 (+ 185us) server_negotiation.cc:407] Connection header received
0927 21:05:58.003867 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:58.003869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:58.003874 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:58.003930 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:58.004754 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:58.004755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:58.004756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:05:58.004761 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:58.004778 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:05:58.004782 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:58.004912 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:58.005067 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 21:05:58.005081 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 21:05:58.006697 5202 catalog_manager.cc:1347] Generated new cluster ID: b33a0bef8f5942cfb1f87982b67a1146
I20260927 21:05:58.006752 5202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:05:58.014334 5202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:58.014557 5202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:58.031523 5202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 291c2e02caca419694ba848ebc27d7c5: Generated new TSK 0
I20260927 21:05:58.031816 5202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:58.039788 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5118
2026-09-27T21:05:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:58 dist-test-slave-6c7s krb5kdc[5217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:58 dist-test-slave-6c7s krb5kdc[5217](info): set up 2 sockets
Sep 27 21:05:58 dist-test-slave-6c7s krb5kdc[5217](info): commencing operation
krb5kdc: starting...
W20260927 21:05:59.166051 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:05:59.297101 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:05:59.381872 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:05:59.382088 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:05:59.382113 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:05:59.383363 5230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:05:59.383409 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:05:59.383424 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:05:59.383437 5230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:05:59.383450 5230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:05:59.385041 5230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:05:59.385299 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:05:59.385527 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:05:59.390906 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:59.390848 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:05:59.390849 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:05:59.391091 5230 server_base.cc:1034] running on GCE node
I20260927 21:05:59.391446 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:05:59.391695 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:05:59.392891 5230 hybrid_clock.cc:648] HybridClock initialized: now 1790543159392877 us; error 29 us; skew 500 ppm
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:05:59.395470 5230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:05:59.395857 5230 webserver.cc:460] Webserver started at http://127.0.0.1:36493/ using document root <none> and password file <none>
I20260927 21:05:59.396054 5230 fs_manager.cc:359] Metadata directory not provided
I20260927 21:05:59.396095 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:05:59.396185 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:05:59.397307 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "1901a546a48b4ddaaa9996b95bc0e3d1"
format_stamp: "Formatted at 2026-09-27 21:05:59 on dist-test-slave-6c7s"
server_key: "90366323586ed37755be94ff89609df8"
server_key_iv: "85927607a66ae03b283f4ffd197b2898"
server_key_version: "encryptionkey@0"
I20260927 21:05:59.397636 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "1901a546a48b4ddaaa9996b95bc0e3d1"
format_stamp: "Formatted at 2026-09-27 21:05:59 on dist-test-slave-6c7s"
server_key: "90366323586ed37755be94ff89609df8"
server_key_iv: "85927607a66ae03b283f4ffd197b2898"
server_key_version: "encryptionkey@0"
I20260927 21:05:59.398983 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:05:59.399745 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:05:59.399997 5230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:05:59.400100 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "1901a546a48b4ddaaa9996b95bc0e3d1"
format_stamp: "Formatted at 2026-09-27 21:05:59 on dist-test-slave-6c7s"
server_key: "90366323586ed37755be94ff89609df8"
server_key_iv: "85927607a66ae03b283f4ffd197b2898"
server_key_version: "encryptionkey@0"
I20260927 21:05:59.400180 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:05:59.409812 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:05:59.412302 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:05:59.412508 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:05:59.416455 5230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37181
I20260927 21:05:59.416544 5297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37181 every 8 connection(s)
I20260927 21:05:59.416947 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:05:59.418123 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:05:59.420717 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1: Bootstrap starting.
I20260927 21:05:59.421358 5298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1: Neither blocks nor log segments found. Creating new log.
I20260927 21:05:59.421607 5298 log.cc:824] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1: Log is configured to *not* fsync() on all Append() calls
I20260927 21:05:59.422298 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5230
I20260927 21:05:59.422429 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:05:59.422559 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1: No bootstrap required, opened a new log
I20260927 21:05:59.422644 17147 external_mini_cluster.cc:1428] Setting key ba1c49097244f95d7f94bed5a34ab7d2
I20260927 21:05:59.425307 5298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260927 21:05:59.425522 5298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:05:59.425576 5298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1901a546a48b4ddaaa9996b95bc0e3d1, State: Initialized, Role: FOLLOWER
I20260927 21:05:59.425737 5298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260927 21:05:59.425830 5298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:05:59.425879 5298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:05:59.425954 5298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:05:59.426632 5298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260927 21:05:59.426790 5298 leader_election.cc:302] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1901a546a48b4ddaaa9996b95bc0e3d1; no voters:
I20260927 21:05:59.426993 5298 leader_election.cc:288] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:05:59.427024 5303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:05:59.427197 5303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [term 1 LEADER]: Becoming Leader. State: Replica: 1901a546a48b4ddaaa9996b95bc0e3d1, State: Running, Role: LEADER
I20260927 21:05:59.427297 5303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260927 21:05:59.427284 5298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:05:59.427745 5304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1901a546a48b4ddaaa9996b95bc0e3d1. Latest consensus state: current_term: 1 leader_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } } }
I20260927 21:05:59.427830 5304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:59.427729 5305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1901a546a48b4ddaaa9996b95bc0e3d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } } }
I20260927 21:05:59.427958 5305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1 [sys.catalog]: This master's current role is: LEADER
I20260927 21:05:59.428726 5307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:05:59.429278 5307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:05:59.431542 5307 catalog_manager.cc:1347] Generated new cluster ID: aa3e1b4716664b69b95385b8b761db8e
I20260927 21:05:59.431622 5307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:05:59.435981 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:59.423283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33067 (local address 127.0.0.1:37181)
0927 21:05:59.423466 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:59.423476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:59.423513 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:05:59.423984 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:59.423992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:59.424041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:59.424161 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:59.425155 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:59.426605 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:59.427834 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:59.428599 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:59.428707 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:59.432450 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:59.432454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:59.432455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:59.432468 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:59.434311 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:59.434731 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:59.434742 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:59.434743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:59.434840 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:59.435059 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:59.435060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:59.435061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:59.435321 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:59.435491 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:59.435609 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 21:05:59.435770 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260927 21:05:59.436997 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:05:59.439203 5307 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:05:59.439903 5307 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:05:59.444715 5302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:05:59.438490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57903 (local address 127.0.0.1:37181)
0927 21:05:59.438528 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:05:59.438530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:05:59.438539 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:05:59.438594 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:05:59.438595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:05:59.438604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:05:59.438658 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:05:59.439369 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:59.439858 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:59.440590 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:05:59.440805 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:05:59.440909 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:05:59.442134 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:05:59.442136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:05:59.442136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:05:59.442143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:05:59.443567 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:59.443919 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:59.443920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:59.443920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:59.443975 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:05:59.444177 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:05:59.444178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:05:59.444178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:05:59.444300 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:05:59.444406 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 21:05:59.444517 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 21:05:59.444605 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:05:59.447450 5307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1901a546a48b4ddaaa9996b95bc0e3d1: Generated new TSK 0
I20260927 21:05:59.447659 5307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:05:59.455147 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5230
2026-09-27T21:05:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5329](info): set up 2 sockets
Sep 27 21:05:59 dist-test-slave-6c7s krb5kdc[5329](info): commencing operation
krb5kdc: starting...
W20260927 21:06:00.513891 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:00 dist-test-slave-6c7s krb5kdc[5329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:00.645246 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:00.729175 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:00.729400 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:00.729425 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:00.730602 5342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:00.730644 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:00.730659 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:00.730672 5342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:00.730687 5342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:00.732137 5342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:00.732354 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:00.732533 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:00.737839 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:00.737864 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:00.737864 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:00.738044 5342 server_base.cc:1034] running on GCE node
I20260927 21:06:00.738440 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:00.738742 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:00.739992 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790543160739990 us; error 53 us; skew 500 ppm
Sep 27 21:06:00 dist-test-slave-6c7s krb5kdc[5329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:00.742851 5342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:00.743311 5342 webserver.cc:460] Webserver started at http://127.0.0.1:45957/ using document root <none> and password file <none>
I20260927 21:06:00.743577 5342 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:00.743651 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:00.743804 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:00.744992 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "ae223b83c6744f3882b1cd3de255cbf3"
format_stamp: "Formatted at 2026-09-27 21:06:00 on dist-test-slave-6c7s"
server_key: "722daff0aabf16ad704fa95e36eb0422"
server_key_iv: "36aef0ab7c982deb53ea503c02896f41"
server_key_version: "encryptionkey@0"
I20260927 21:06:00.745371 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "ae223b83c6744f3882b1cd3de255cbf3"
format_stamp: "Formatted at 2026-09-27 21:06:00 on dist-test-slave-6c7s"
server_key: "722daff0aabf16ad704fa95e36eb0422"
server_key_iv: "36aef0ab7c982deb53ea503c02896f41"
server_key_version: "encryptionkey@0"
I20260927 21:06:00.746812 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 21:06:00.747609 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:00.747802 5342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:00.747911 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "ae223b83c6744f3882b1cd3de255cbf3"
format_stamp: "Formatted at 2026-09-27 21:06:00 on dist-test-slave-6c7s"
server_key: "722daff0aabf16ad704fa95e36eb0422"
server_key_iv: "36aef0ab7c982deb53ea503c02896f41"
server_key_version: "encryptionkey@0"
I20260927 21:06:00.748003 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:00.770160 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:00.772338 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:00.772511 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:00.776716 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38049
I20260927 21:06:00.776758 5410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38049 every 8 connection(s)
I20260927 21:06:00.777199 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:00.778157 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:00.780622 5411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: Bootstrap starting.
I20260927 21:06:00.780804 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5342
I20260927 21:06:00.780961 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:00.781191 17147 external_mini_cluster.cc:1428] Setting key 580785da80953c875a6583741cc12e08
I20260927 21:06:00.781447 5411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:00.781831 5411 log.cc:824] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:00.782549 5411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: No bootstrap required, opened a new log
I20260927 21:06:00.784420 5411 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } }
I20260927 21:06:00.784590 5411 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:00.784637 5411 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae223b83c6744f3882b1cd3de255cbf3, State: Initialized, Role: FOLLOWER
I20260927 21:06:00.784781 5411 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } }
I20260927 21:06:00.784910 5411 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:00.784950 5411 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:00.785003 5411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:00.785576 5411 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } }
I20260927 21:06:00.785704 5411 leader_election.cc:302] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae223b83c6744f3882b1cd3de255cbf3; no voters:
I20260927 21:06:00.785881 5411 leader_election.cc:288] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:00.785917 5416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:00.786032 5416 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [term 1 LEADER]: Becoming Leader. State: Replica: ae223b83c6744f3882b1cd3de255cbf3, State: Running, Role: LEADER
I20260927 21:06:00.786118 5416 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } }
I20260927 21:06:00.786149 5411 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:00.786453 5416 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae223b83c6744f3882b1cd3de255cbf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } } }
I20260927 21:06:00.786525 5416 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:00.786654 5418 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae223b83c6744f3882b1cd3de255cbf3. Latest consensus state: current_term: 1 leader_uuid: "ae223b83c6744f3882b1cd3de255cbf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae223b83c6744f3882b1cd3de255cbf3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38049 } } }
I20260927 21:06:00.786957 5418 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:00.787721 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:00.781775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52837 (local address 127.0.0.1:38049)
0927 21:06:00.782006 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:00.782017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:00.782061 (+ 44us) server_negotiation.cc:407] Connection header received
0927 21:06:00.782673 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:00.782684 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:00.782761 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:00.782939 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:00.783166 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:00.784151 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:00.785326 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:00.785911 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:00.786091 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:00.786877 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:00.786881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:00.786882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:00.787115 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:00.787149 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:00.787161 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:00.787257 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:00.787419 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 21:06:00.787505 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260927 21:06:00.788081 5431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:00.788169 5431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:00.788234 5421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:00.788403 5421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:00.788920 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:00.790207 5421 catalog_manager.cc:1347] Generated new cluster ID: 92b704a0fbc94f10b18a2a393fd969dc
I20260927 21:06:00.790271 5421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:00.793550 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:00.790364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60405 (local address 127.0.0.1:38049)
0927 21:06:00.790403 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:00.790405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:00.790570 (+ 165us) server_negotiation.cc:407] Connection header received
0927 21:06:00.790632 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:00.790633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:00.790644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:00.790699 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:00.790968 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:00.791456 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:00.792202 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:00.792420 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:00.792498 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:00.793273 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:00.793275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:00.793276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:00.793281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:00.793296 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:00.793300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:00.793331 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:00.793448 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 21:06:00.793475 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:06:00.806967 5421 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:00.807607 5421 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:00.812927 5421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae223b83c6744f3882b1cd3de255cbf3: Generated new TSK 0
I20260927 21:06:00.813148 5421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:00.822335 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5342
2026-09-27T21:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:00 dist-test-slave-6c7s krb5kdc[5442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:00 dist-test-slave-6c7s krb5kdc[5442](info): set up 2 sockets
Sep 27 21:06:00 dist-test-slave-6c7s krb5kdc[5442](info): commencing operation
krb5kdc: starting...
W20260927 21:06:01.881814 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:01 dist-test-slave-6c7s krb5kdc[5442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:02.013329 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:02.100400 5455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:02.100673 5455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:02.100726 5455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:02.102021 5455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:02.102104 5455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:02.102128 5455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:02.102150 5455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:02.102178 5455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:02.103763 5455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:02.104075 5455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:02.104323 5455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:02.109666 5461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:02.109679 5460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:02.109692 5463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:02.110435 5455 server_base.cc:1034] running on GCE node
I20260927 21:06:02.110754 5455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:02.111080 5455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:02.112257 5455 hybrid_clock.cc:648] HybridClock initialized: now 1790543162112248 us; error 64 us; skew 500 ppm
Sep 27 21:06:02 dist-test-slave-6c7s krb5kdc[5442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:02.119760 5455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:02.120282 5455 webserver.cc:460] Webserver started at http://127.0.0.1:34121/ using document root <none> and password file <none>
I20260927 21:06:02.120553 5455 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:02.120625 5455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:02.120784 5455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:02.121939 5455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "8f0fee74cf734c759d92919449483179"
format_stamp: "Formatted at 2026-09-27 21:06:02 on dist-test-slave-6c7s"
server_key: "69e71e0f89d8e14d1280f4ad1d5c3ca3"
server_key_iv: "0710ec1f26187d4a61e98a9586c7da09"
server_key_version: "encryptionkey@0"
I20260927 21:06:02.122308 5455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "8f0fee74cf734c759d92919449483179"
format_stamp: "Formatted at 2026-09-27 21:06:02 on dist-test-slave-6c7s"
server_key: "69e71e0f89d8e14d1280f4ad1d5c3ca3"
server_key_iv: "0710ec1f26187d4a61e98a9586c7da09"
server_key_version: "encryptionkey@0"
I20260927 21:06:02.123951 5455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:06:02.124939 5470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:02.125131 5455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:02.125239 5455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "8f0fee74cf734c759d92919449483179"
format_stamp: "Formatted at 2026-09-27 21:06:02 on dist-test-slave-6c7s"
server_key: "69e71e0f89d8e14d1280f4ad1d5c3ca3"
server_key_iv: "0710ec1f26187d4a61e98a9586c7da09"
server_key_version: "encryptionkey@0"
I20260927 21:06:02.125326 5455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:02.128911 5455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:02.131206 5455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:02.131414 5455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:02.136134 5455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35547
I20260927 21:06:02.136324 5522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35547 every 8 connection(s)
I20260927 21:06:02.136602 5455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:02.138242 5523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:02.138550 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5455
I20260927 21:06:02.138691 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:02.138881 17147 external_mini_cluster.cc:1428] Setting key 43cd3425a3f2cb6738aade8737761689
I20260927 21:06:02.141566 5523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179: Bootstrap starting.
I20260927 21:06:02.141949 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:02.139501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53717 (local address 127.0.0.1:35547)
0927 21:06:02.139736 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:02.139746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:02.139782 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:06:02.140328 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:02.140337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:02.140375 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:02.140608 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:02.141318 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:02.141322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:02.141322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:02.141327 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:02.141371 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:02.141387 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:02.141468 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:02.141715 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 21:06:02.141759 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260927 21:06:02.142305 5523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:02.142642 5523 log.cc:824] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:02.143122 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:02.143440 5523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179: No bootstrap required, opened a new log
I20260927 21:06:02.146358 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:02.144680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43005 (local address 127.0.0.1:35547)
0927 21:06:02.144724 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:02.144726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:02.144923 (+ 197us) server_negotiation.cc:407] Connection header received
0927 21:06:02.144999 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:02.145000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:02.145004 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:02.145052 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:02.145788 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:02.145789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:02.145789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:02.145816 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:02.145838 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:02.145842 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:02.145897 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:02.146216 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 21:06:02.146232 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 21:06:02.146097 5523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260927 21:06:02.146516 5523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:02.146564 5523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0fee74cf734c759d92919449483179, State: Initialized, Role: FOLLOWER
I20260927 21:06:02.146687 5523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260927 21:06:02.146757 5523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:02.146781 5523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:02.146832 5523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:02.147456 5523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260927 21:06:02.147545 5523 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0fee74cf734c759d92919449483179; no voters:
I20260927 21:06:02.147729 5523 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:02.147830 5533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:02.148000 5533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [term 1 LEADER]: Becoming Leader. State: Replica: 8f0fee74cf734c759d92919449483179, State: Running, Role: LEADER
I20260927 21:06:02.148018 5523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:02.148139 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260927 21:06:02.148509 5533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0fee74cf734c759d92919449483179. Latest consensus state: current_term: 1 leader_uuid: "8f0fee74cf734c759d92919449483179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260927 21:06:02.148541 5534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0fee74cf734c759d92919449483179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0fee74cf734c759d92919449483179" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260927 21:06:02.148769 5534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:02.149042 5542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:02.149204 5533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:02.149618 5542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:02.151417 5542 catalog_manager.cc:1347] Generated new cluster ID: e8f9b6be34a44dae840024eda84b66bd
I20260927 21:06:02.151474 5542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:02.156656 5542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:02.156872 5542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:02.163111 5542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0fee74cf734c759d92919449483179: Generated new TSK 0
I20260927 21:06:02.163367 5542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:02.172703 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5455
2026-09-27T21:06:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:02 dist-test-slave-6c7s krb5kdc[5554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:02 dist-test-slave-6c7s krb5kdc[5554](info): set up 2 sockets
Sep 27 21:06:02 dist-test-slave-6c7s krb5kdc[5554](info): commencing operation
krb5kdc: starting...
W20260927 21:06:03.229830 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:03.357223 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:03.442742 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:03.443027 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:03.443068 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:03.444334 5567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:03.444437 5567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:03.444463 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:03.444499 5567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:03.444540 5567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:03.446293 5567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:03.446621 5567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:03.446882 5567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:03.452083 5572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:03.452361 5567 server_base.cc:1034] running on GCE node
W20260927 21:06:03.452128 5573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:03.452086 5575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:03.452762 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:03.453150 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:03.454330 5567 hybrid_clock.cc:648] HybridClock initialized: now 1790543163454312 us; error 45 us; skew 500 ppm
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:03.457573 5567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:03.458271 5567 webserver.cc:460] Webserver started at http://127.0.0.1:42135/ using document root <none> and password file <none>
I20260927 21:06:03.458525 5567 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:03.458573 5567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:03.458726 5567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:03.459895 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "a8c10114c85940069af1cdbefe84f95d"
format_stamp: "Formatted at 2026-09-27 21:06:03 on dist-test-slave-6c7s"
server_key: "8954d1e1fceeb18193f76b2e3f201bd0"
server_key_iv: "bee97a0dbdb873837b3d4d5f164ad6c2"
server_key_version: "encryptionkey@0"
I20260927 21:06:03.460287 5567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "a8c10114c85940069af1cdbefe84f95d"
format_stamp: "Formatted at 2026-09-27 21:06:03 on dist-test-slave-6c7s"
server_key: "8954d1e1fceeb18193f76b2e3f201bd0"
server_key_iv: "bee97a0dbdb873837b3d4d5f164ad6c2"
server_key_version: "encryptionkey@0"
I20260927 21:06:03.461891 5567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:03.462868 5582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:03.463194 5567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:03.463316 5567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "a8c10114c85940069af1cdbefe84f95d"
format_stamp: "Formatted at 2026-09-27 21:06:03 on dist-test-slave-6c7s"
server_key: "8954d1e1fceeb18193f76b2e3f201bd0"
server_key_iv: "bee97a0dbdb873837b3d4d5f164ad6c2"
server_key_version: "encryptionkey@0"
I20260927 21:06:03.463403 5567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:03.494313 5567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:03.496740 5567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:03.496951 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:03.501459 5567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35961
I20260927 21:06:03.501547 5635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35961 every 8 connection(s)
I20260927 21:06:03.501925 5567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:03.502731 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5567
I20260927 21:06:03.502859 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:03.503026 5636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:03.503090 17147 external_mini_cluster.cc:1428] Setting key a37efbcbd6c49babb9dd4104150a31fa
I20260927 21:06:03.506382 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d: Bootstrap starting.
I20260927 21:06:03.507081 5636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:03.507385 5636 log.cc:824] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:03.508153 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d: No bootstrap required, opened a new log
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:06:03.510826 5636 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } }
I20260927 21:06:03.510980 5636 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:03.511003 5636 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c10114c85940069af1cdbefe84f95d, State: Initialized, Role: FOLLOWER
I20260927 21:06:03.511144 5636 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } }
I20260927 21:06:03.511206 5636 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:03.511228 5636 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:03.511293 5636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:03.511880 5636 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } }
I20260927 21:06:03.512001 5636 leader_election.cc:302] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c10114c85940069af1cdbefe84f95d; no voters:
I20260927 21:06:03.512207 5636 leader_election.cc:288] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:03.512243 5641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:03.512342 5641 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [term 1 LEADER]: Becoming Leader. State: Replica: a8c10114c85940069af1cdbefe84f95d, State: Running, Role: LEADER
I20260927 21:06:03.512454 5641 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } }
I20260927 21:06:03.512480 5636 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:03.513445 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c10114c85940069af1cdbefe84f95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } } }
I20260927 21:06:03.513471 5643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c10114c85940069af1cdbefe84f95d. Latest consensus state: current_term: 1 leader_uuid: "a8c10114c85940069af1cdbefe84f95d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c10114c85940069af1cdbefe84f95d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35961 } } }
I20260927 21:06:03.513545 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:03.513561 5643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:03.513859 5653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:03.514358 5653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:03.515769 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:03.503668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:34503 (local address 127.0.0.1:35961)
0927 21:06:03.503861 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:03.503871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:03.503914 (+ 43us) server_negotiation.cc:407] Connection header received
0927 21:06:03.504554 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:03.504562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:03.504617 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:03.504757 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:03.505653 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:03.506727 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:03.507794 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:03.508324 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:03.508414 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:03.511384 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:03.511390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:03.511390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:03.511404 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:03.513759 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:03.514360 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:03.514364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:03.514365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:03.514478 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:03.514741 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:03.514743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:03.514743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:03.515061 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:03.515258 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:03.515448 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 21:06:03.515567 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260927 21:06:03.516423 5653 catalog_manager.cc:1347] Generated new cluster ID: bc3a0eb7499a4a0eb136a974a0e79f15
I20260927 21:06:03.516479 5653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:03.516688 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:03.525051 5653 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:03.525597 5653 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:03.527379 5653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c10114c85940069af1cdbefe84f95d: Generated new TSK 0
I20260927 21:06:03.527565 5653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:03.530951 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:03.521262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:43591 (local address 127.0.0.1:35961)
0927 21:06:03.521311 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:03.521313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:03.521474 (+ 161us) server_negotiation.cc:407] Connection header received
0927 21:06:03.521541 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:03.521542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:03.521552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:03.521608 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:03.522332 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:03.523112 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:03.524190 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:03.524489 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:03.524560 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:03.528170 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:03.528171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:03.528172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:03.528179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:03.529761 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:03.530198 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:03.530200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:03.530200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:03.530259 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:03.530494 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:03.530495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:03.530495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:03.530593 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:03.530667 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:03.530836 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 21:06:03.530891 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 21:06:03.533205 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5567
W20260927 21:06:03.537379 5552 connection.cc:535] client connection to 127.0.0.1:35961 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35961): Connection reset by peer (error 104)
2026-09-27T21:06:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5667](info): set up 2 sockets
Sep 27 21:06:03 dist-test-slave-6c7s krb5kdc[5667](info): commencing operation
krb5kdc: starting...
W20260927 21:06:04.573908 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:04 dist-test-slave-6c7s krb5kdc[5667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:04.696165 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:04.777968 5681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:04.778193 5681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:04.778225 5681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:04.779418 5681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:04.779469 5681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:04.779484 5681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:04.779496 5681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:04.779510 5681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:04.781121 5681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:04.781409 5681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:04.781646 5681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:04.786825 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:04.786825 5687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:04.786965 5681 server_base.cc:1034] running on GCE node
W20260927 21:06:04.786828 5689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:04.787405 5681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:04.787806 5681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:04.789033 5681 hybrid_clock.cc:648] HybridClock initialized: now 1790543164789005 us; error 36 us; skew 500 ppm
Sep 27 21:06:04 dist-test-slave-6c7s krb5kdc[5667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:04.791955 5681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:04.792486 5681 webserver.cc:460] Webserver started at http://127.0.0.1:41025/ using document root <none> and password file <none>
I20260927 21:06:04.792733 5681 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:04.792806 5681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:04.792989 5681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:04.794104 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "1de028cfa7074462914e70dd2e1cffd4"
format_stamp: "Formatted at 2026-09-27 21:06:04 on dist-test-slave-6c7s"
server_key: "beca325b1a5af7c7a6096ad1d4c5d95d"
server_key_iv: "c82676b036b4d73c36b3895ed6a036fb"
server_key_version: "encryptionkey@0"
I20260927 21:06:04.794492 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "1de028cfa7074462914e70dd2e1cffd4"
format_stamp: "Formatted at 2026-09-27 21:06:04 on dist-test-slave-6c7s"
server_key: "beca325b1a5af7c7a6096ad1d4c5d95d"
server_key_iv: "c82676b036b4d73c36b3895ed6a036fb"
server_key_version: "encryptionkey@0"
I20260927 21:06:04.795938 5681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:04.796890 5696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:04.797088 5681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:04.797200 5681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "1de028cfa7074462914e70dd2e1cffd4"
format_stamp: "Formatted at 2026-09-27 21:06:04 on dist-test-slave-6c7s"
server_key: "beca325b1a5af7c7a6096ad1d4c5d95d"
server_key_iv: "c82676b036b4d73c36b3895ed6a036fb"
server_key_version: "encryptionkey@0"
I20260927 21:06:04.797295 5681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:04.832268 5681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:04.834830 5681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:04.835031 5681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:04.839188 5681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42397
I20260927 21:06:04.839218 5748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42397 every 8 connection(s)
I20260927 21:06:04.839588 5681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:04.840463 5749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:04.841557 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5681
I20260927 21:06:04.841703 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:04.841915 17147 external_mini_cluster.cc:1428] Setting key 94e018713070dded8c2340fbfeeff377
I20260927 21:06:04.843245 5749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4: Bootstrap starting.
I20260927 21:06:04.843863 5749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:04.844147 5749 log.cc:824] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:04.844882 5749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4: No bootstrap required, opened a new log
I20260927 21:06:04.846642 5749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260927 21:06:04.846805 5749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:04.846856 5749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de028cfa7074462914e70dd2e1cffd4, State: Initialized, Role: FOLLOWER
I20260927 21:06:04.847033 5749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260927 21:06:04.847136 5749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:04.847178 5749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:04.847227 5749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:04.847518 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:04.842498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:34575 (local address 127.0.0.1:42397)
0927 21:06:04.842668 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:04.842677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:04.842706 (+ 29us) server_negotiation.cc:407] Connection header received
0927 21:06:04.843116 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:04.843122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:04.843224 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:04.843394 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:04.843627 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:04.844619 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:04.845524 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:04.845945 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:04.846101 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:04.846834 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:04.846839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:04.846839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:04.847057 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:04.847088 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:04.847099 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:04.847197 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:04.847313 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 21:06:04.847345 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260927 21:06:04.847982 5749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260927 21:06:04.848090 5749 leader_election.cc:302] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de028cfa7074462914e70dd2e1cffd4; no voters:
I20260927 21:06:04.848250 5749 leader_election.cc:288] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:04.848312 5754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:04.848476 5754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1de028cfa7074462914e70dd2e1cffd4, State: Running, Role: LEADER
I20260927 21:06:04.848516 5749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:04.848589 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:04.848624 5754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } }
I20260927 21:06:04.848996 5754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de028cfa7074462914e70dd2e1cffd4. Latest consensus state: current_term: 1 leader_uuid: "1de028cfa7074462914e70dd2e1cffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } } }
I20260927 21:06:04.849085 5756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de028cfa7074462914e70dd2e1cffd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de028cfa7074462914e70dd2e1cffd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42397 } } }
I20260927 21:06:04.849112 5754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:04.849143 5756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:04.849375 5762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:04.849920 5762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:04.851898 5762 catalog_manager.cc:1347] Generated new cluster ID: 8d32099100b747bebc532752c34553cc
I20260927 21:06:04.851951 5762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:04.854179 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:04.850657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:36055 (local address 127.0.0.1:42397)
0927 21:06:04.850698 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:04.850700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:04.850823 (+ 123us) server_negotiation.cc:407] Connection header received
0927 21:06:04.850868 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:04.850869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:04.850881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:04.850943 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:04.851076 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:04.851740 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:04.852750 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:04.853092 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:04.853172 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:04.853924 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:04.853925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:04.853925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:04.853930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:04.853946 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:04.853951 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:04.854018 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:04.854090 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 21:06:04.854104 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 21:06:04.857394 5762 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:04.857966 5762 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:04.862788 5762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de028cfa7074462914e70dd2e1cffd4: Generated new TSK 0
I20260927 21:06:04.863001 5762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:04.871039 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5681
W20260927 21:06:04.875437 5665 connection.cc:535] client connection to 127.0.0.1:42397 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42397): Connection reset by peer (error 104)
2026-09-27T21:06:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:04 dist-test-slave-6c7s krb5kdc[5781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:04 dist-test-slave-6c7s krb5kdc[5781](info): set up 2 sockets
Sep 27 21:06:04 dist-test-slave-6c7s krb5kdc[5781](info): commencing operation
krb5kdc: starting...
W20260927 21:06:05.909858 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:05 dist-test-slave-6c7s krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:06.041877 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:06.125036 5794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:06.125262 5794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:06.125285 5794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:06.126492 5794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:06.126539 5794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:06.126554 5794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:06.126566 5794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:06.126609 5794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:06.128165 5794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:06.128444 5794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:06.128683 5794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:06.133744 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:06.133773 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:06.133775 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:06.134315 5794 server_base.cc:1034] running on GCE node
I20260927 21:06:06.134562 5794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:06.134857 5794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:06.136020 5794 hybrid_clock.cc:648] HybridClock initialized: now 1790543166135995 us; error 43 us; skew 500 ppm
Sep 27 21:06:06 dist-test-slave-6c7s krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:06.138759 5794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:06.139225 5794 webserver.cc:460] Webserver started at http://127.0.0.1:46549/ using document root <none> and password file <none>
I20260927 21:06:06.139430 5794 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:06.139472 5794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:06.139565 5794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:06.140630 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "78b321ebb2f94f8a84f58e41b50cf2ed"
format_stamp: "Formatted at 2026-09-27 21:06:06 on dist-test-slave-6c7s"
server_key: "30dc64614538724fe85e6f68dc43cb31"
server_key_iv: "6083b5b846f86e07c3a33215730676b5"
server_key_version: "encryptionkey@0"
I20260927 21:06:06.141124 5794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "78b321ebb2f94f8a84f58e41b50cf2ed"
format_stamp: "Formatted at 2026-09-27 21:06:06 on dist-test-slave-6c7s"
server_key: "30dc64614538724fe85e6f68dc43cb31"
server_key_iv: "6083b5b846f86e07c3a33215730676b5"
server_key_version: "encryptionkey@0"
I20260927 21:06:06.142532 5794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:06.143477 5809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:06.143754 5794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:06.143832 5794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "78b321ebb2f94f8a84f58e41b50cf2ed"
format_stamp: "Formatted at 2026-09-27 21:06:06 on dist-test-slave-6c7s"
server_key: "30dc64614538724fe85e6f68dc43cb31"
server_key_iv: "6083b5b846f86e07c3a33215730676b5"
server_key_version: "encryptionkey@0"
I20260927 21:06:06.143893 5794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:06.147344 5794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:06.150484 5794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:06.150653 5794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:06.155648 5794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34125
I20260927 21:06:06.155721 5861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34125 every 8 connection(s)
I20260927 21:06:06.156104 5794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:06.156997 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5794
I20260927 21:06:06.157159 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:06.157297 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:06.157429 17147 external_mini_cluster.cc:1428] Setting key 1af64e4b6f125865c2744542f669e11b
I20260927 21:06:06.160159 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:06.158011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:55691 (local address 127.0.0.1:34125)
0927 21:06:06.158224 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:06.158231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:06.158257 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:06:06.158815 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:06.158822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:06.158850 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:06.159010 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:06.159742 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:06.159746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:06.159746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:06.159751 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:06.159795 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:06.159805 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:06.159886 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:06.159953 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 21:06:06.159996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260927 21:06:06.160822 5862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed: Bootstrap starting.
I20260927 21:06:06.161369 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:06.161669 5862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:06.162030 5862 log.cc:824] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:06.162914 5862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed: No bootstrap required, opened a new log
I20260927 21:06:06.164258 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:06.163041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:53755 (local address 127.0.0.1:34125)
0927 21:06:06.163096 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:06.163099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:06.163247 (+ 148us) server_negotiation.cc:407] Connection header received
0927 21:06:06.163313 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:06.163313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:06.163318 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:06.163361 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:06.163978 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:06.163979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:06.163980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:06.163983 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:06.163994 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:06.163997 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:06.164043 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:06.164104 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 21:06:06.164114 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 21:06:06.165624 5862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } }
I20260927 21:06:06.165813 5862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:06.165867 5862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b321ebb2f94f8a84f58e41b50cf2ed, State: Initialized, Role: FOLLOWER
I20260927 21:06:06.166007 5862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } }
I20260927 21:06:06.166111 5862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:06.166159 5862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:06.166216 5862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:06.166832 5862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } }
I20260927 21:06:06.166961 5862 leader_election.cc:302] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78b321ebb2f94f8a84f58e41b50cf2ed; no voters:
I20260927 21:06:06.167156 5862 leader_election.cc:288] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:06.167192 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:06.167423 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [term 1 LEADER]: Becoming Leader. State: Replica: 78b321ebb2f94f8a84f58e41b50cf2ed, State: Running, Role: LEADER
I20260927 21:06:06.167511 5862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:06.167533 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } }
I20260927 21:06:06.167866 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b321ebb2f94f8a84f58e41b50cf2ed. Latest consensus state: current_term: 1 leader_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } } }
I20260927 21:06:06.167950 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:06.167969 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b321ebb2f94f8a84f58e41b50cf2ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34125 } } }
I20260927 21:06:06.168075 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:06.169040 5878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:06.169821 5878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:06.171379 5878 catalog_manager.cc:1347] Generated new cluster ID: 26d5d1d2885d42ddb9fa79dc6f555c66
I20260927 21:06:06.171435 5878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:06.183969 5878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:06.184231 5878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:06.190477 5878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78b321ebb2f94f8a84f58e41b50cf2ed: Generated new TSK 0
I20260927 21:06:06.190764 5878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:06.198127 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5794
2026-09-27T21:06:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:06 dist-test-slave-6c7s krb5kdc[5893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:06 dist-test-slave-6c7s krb5kdc[5893](info): set up 2 sockets
Sep 27 21:06:06 dist-test-slave-6c7s krb5kdc[5893](info): commencing operation
krb5kdc: starting...
W20260927 21:06:07.257756 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[5893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:07.384755 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:07.470726 5907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:07.470991 5907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:07.471033 5907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:07.472223 5907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:07.472313 5907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:07.472337 5907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:07.472358 5907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:07.472388 5907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:07.474023 5907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:07.474345 5907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:07.474594 5907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:07.479694 5915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:07.479700 5912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:07.479715 5913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:07.479959 5907 server_base.cc:1034] running on GCE node
I20260927 21:06:07.480197 5907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:07.480458 5907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:07.481629 5907 hybrid_clock.cc:648] HybridClock initialized: now 1790543167481598 us; error 59 us; skew 500 ppm
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[5893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:07.491014 5907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:07.491499 5907 webserver.cc:460] Webserver started at http://127.0.0.1:40941/ using document root <none> and password file <none>
I20260927 21:06:07.491755 5907 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:07.491801 5907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:07.491977 5907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:07.493171 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "07c1932af69844538b2e62986e2fc253"
format_stamp: "Formatted at 2026-09-27 21:06:07 on dist-test-slave-6c7s"
server_key: "ed15bfc0e0a2ce39bbd70d601fe25887"
server_key_iv: "925c6d09f777ece7e3e76efe56c29d52"
server_key_version: "encryptionkey@0"
I20260927 21:06:07.493539 5907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "07c1932af69844538b2e62986e2fc253"
format_stamp: "Formatted at 2026-09-27 21:06:07 on dist-test-slave-6c7s"
server_key: "ed15bfc0e0a2ce39bbd70d601fe25887"
server_key_iv: "925c6d09f777ece7e3e76efe56c29d52"
server_key_version: "encryptionkey@0"
I20260927 21:06:07.495018 5907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:06:07.496060 5922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:07.496332 5907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:07.496452 5907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "07c1932af69844538b2e62986e2fc253"
format_stamp: "Formatted at 2026-09-27 21:06:07 on dist-test-slave-6c7s"
server_key: "ed15bfc0e0a2ce39bbd70d601fe25887"
server_key_iv: "925c6d09f777ece7e3e76efe56c29d52"
server_key_version: "encryptionkey@0"
I20260927 21:06:07.496536 5907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:07.516206 5907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:07.518597 5907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:07.518798 5907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:07.522944 5907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33957
I20260927 21:06:07.522981 5974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33957 every 8 connection(s)
I20260927 21:06:07.523468 5907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:07.524590 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:07.527004 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253: Bootstrap starting.
I20260927 21:06:07.527554 5975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:07.527848 5975 log.cc:824] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:07.528569 5975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253: No bootstrap required, opened a new log
I20260927 21:06:07.530259 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 5907
I20260927 21:06:07.530381 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:07.530381 5975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260927 21:06:07.530562 5975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:07.530580 17147 external_mini_cluster.cc:1428] Setting key c73f95eaca88e41391fd274a35c872ad
I20260927 21:06:07.530608 5975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1932af69844538b2e62986e2fc253, State: Initialized, Role: FOLLOWER
I20260927 21:06:07.530774 5975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260927 21:06:07.530889 5975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:07.530929 5975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:07.530987 5975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:07.531713 5975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260927 21:06:07.531810 5975 leader_election.cc:302] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07c1932af69844538b2e62986e2fc253; no voters:
I20260927 21:06:07.532019 5975 leader_election.cc:288] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:07.532040 5980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:07.532153 5980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [term 1 LEADER]: Becoming Leader. State: Replica: 07c1932af69844538b2e62986e2fc253, State: Running, Role: LEADER
I20260927 21:06:07.532312 5975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:07.532292 5980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } }
I20260927 21:06:07.532790 5981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07c1932af69844538b2e62986e2fc253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260927 21:06:07.532840 5982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07c1932af69844538b2e62986e2fc253. Latest consensus state: current_term: 1 leader_uuid: "07c1932af69844538b2e62986e2fc253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07c1932af69844538b2e62986e2fc253" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33957 } } }
I20260927 21:06:07.532974 5981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:07.533185 5982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:07.533362 5989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:07.534053 5989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:07.535904 5989 catalog_manager.cc:1347] Generated new cluster ID: 12e0de0ed4074105a0f0f5acc3307534
I20260927 21:06:07.535953 5989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[5893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 21:06:07.543321 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:07.531091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:46021 (local address 127.0.0.1:33957)
0927 21:06:07.531265 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:07.531275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:07.531315 (+ 40us) server_negotiation.cc:407] Connection header received
0927 21:06:07.531668 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:07.531675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:07.531723 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:07.531891 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:07.532774 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:07.534233 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:07.535231 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:07.535853 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:07.535966 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:07.538955 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:07.538960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:07.538960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:07.538974 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:07.541506 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:07.541986 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:07.541989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:07.541990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:07.542108 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:07.542366 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:07.542366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:07.542367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:07.542670 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:07.542831 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:07.542976 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 21:06:07.543079 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260927 21:06:07.544353 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:07.553746 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:07.546050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:40759 (local address 127.0.0.1:33957)
0927 21:06:07.546083 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:07.546084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:07.546093 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:06:07.546159 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:07.546160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:07.546169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:07.546219 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:07.546888 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:07.547636 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:07.548709 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:07.549047 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:07.549107 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:07.550564 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:07.550565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:07.550566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:07.550572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:07.552147 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:07.552607 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:07.552608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:07.552609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:07.552662 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:07.552909 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:07.552910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:07.552910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:07.553038 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:07.553128 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:07.553512 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 21:06:07.553589 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":16}
I20260927 21:06:07.555755 5989 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:07.556305 5989 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:07.562974 5989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07c1932af69844538b2e62986e2fc253: Generated new TSK 0
I20260927 21:06:07.563217 5989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:07.566264 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 5907
W20260927 21:06:07.570324 5891 connection.cc:535] client connection to 127.0.0.1:33957 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33957): Connection reset by peer (error 104)
2026-09-27T21:06:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[6006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[6006](info): set up 2 sockets
Sep 27 21:06:07 dist-test-slave-6c7s krb5kdc[6006](info): commencing operation
krb5kdc: starting...
W20260927 21:06:08.629819 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:08 dist-test-slave-6c7s krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:08.752496 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:08.853799 6019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:08.854056 6019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:08.854095 6019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:08.855281 6019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:08.855353 6019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:08.855378 6019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:08.855405 6019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:08.855436 6019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:08.857079 6019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:08.857398 6019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:08.857632 6019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:08.862664 6027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:08.862687 6024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:08.862676 6025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:08.862927 6019 server_base.cc:1034] running on GCE node
I20260927 21:06:08.863235 6019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:08.863533 6019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:08.864737 6019 hybrid_clock.cc:648] HybridClock initialized: now 1790543168864722 us; error 29 us; skew 500 ppm
Sep 27 21:06:08 dist-test-slave-6c7s krb5kdc[6006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:08.867440 6019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:08.867898 6019 webserver.cc:460] Webserver started at http://127.0.0.1:40331/ using document root <none> and password file <none>
I20260927 21:06:08.868132 6019 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:08.868201 6019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:08.868355 6019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:08.869534 6019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "d29e6306ecb4424a8b10a1d655482bc9"
format_stamp: "Formatted at 2026-09-27 21:06:08 on dist-test-slave-6c7s"
server_key: "95dae83e8b5265624342eb45d223a541"
server_key_iv: "8d3627bf701eaa3da828bab9490b7b08"
server_key_version: "encryptionkey@0"
I20260927 21:06:08.869925 6019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "d29e6306ecb4424a8b10a1d655482bc9"
format_stamp: "Formatted at 2026-09-27 21:06:08 on dist-test-slave-6c7s"
server_key: "95dae83e8b5265624342eb45d223a541"
server_key_iv: "8d3627bf701eaa3da828bab9490b7b08"
server_key_version: "encryptionkey@0"
I20260927 21:06:08.871322 6019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:06:08.872164 6034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:08.872463 6019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:08.872591 6019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "d29e6306ecb4424a8b10a1d655482bc9"
format_stamp: "Formatted at 2026-09-27 21:06:08 on dist-test-slave-6c7s"
server_key: "95dae83e8b5265624342eb45d223a541"
server_key_iv: "8d3627bf701eaa3da828bab9490b7b08"
server_key_version: "encryptionkey@0"
I20260927 21:06:08.872675 6019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:08.888366 6019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:08.890900 6019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:08.891099 6019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:08.895334 6019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40133
I20260927 21:06:08.895375 6087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40133 every 8 connection(s)
I20260927 21:06:08.895746 6019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:08.896633 6088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:08.897658 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6019
I20260927 21:06:08.897789 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:08.898000 17147 external_mini_cluster.cc:1428] Setting key bff0c214a1784f486968c16ff8098f6b
I20260927 21:06:08.900100 6088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: Bootstrap starting.
I20260927 21:06:08.900748 6088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:08.901105 6088 log.cc:824] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:08.901868 6088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: No bootstrap required, opened a new log
I20260927 21:06:08.903813 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:08.898554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:42653 (local address 127.0.0.1:40133)
0927 21:06:08.898736 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:08.898744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:08.898771 (+ 27us) server_negotiation.cc:407] Connection header received
0927 21:06:08.899190 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:08.899196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:08.899254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:08.899407 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:08.899666 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:08.900631 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:08.901705 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:08.902215 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:08.902307 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:08.903047 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:08.903051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:08.903051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:08.903271 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:08.903305 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:08.903317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:08.903384 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:08.903587 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 21:06:08.903620 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260927 21:06:08.904206 6088 raft_consensus.cc:348] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260927 21:06:08.904386 6088 raft_consensus.cc:374] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:08.904425 6088 raft_consensus.cc:729] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29e6306ecb4424a8b10a1d655482bc9, State: Initialized, Role: FOLLOWER
I20260927 21:06:08.904600 6088 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260927 21:06:08.904696 6088 raft_consensus.cc:388] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:08.904731 6088 raft_consensus.cc:482] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:08.904780 6088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:08.904917 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:08.905587 6088 raft_consensus.cc:504] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260927 21:06:08.905699 6088 leader_election.cc:302] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d29e6306ecb4424a8b10a1d655482bc9; no voters:
I20260927 21:06:08.905892 6088 leader_election.cc:288] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:08.905966 6093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:08.906147 6093 raft_consensus.cc:686] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [term 1 LEADER]: Becoming Leader. State: Replica: d29e6306ecb4424a8b10a1d655482bc9, State: Running, Role: LEADER
I20260927 21:06:08.906289 6093 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } }
I20260927 21:06:08.906292 6088 sys_catalog.cc:564] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:08.907613 6094 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d29e6306ecb4424a8b10a1d655482bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } } }
I20260927 21:06:08.907727 6094 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [sys.catalog]: This master's current role is: LEADER
W20260927 21:06:08.908712 6112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:08.908774 6112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:08.908941 6113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:08.909111 6098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29e6306ecb4424a8b10a1d655482bc9. Latest consensus state: current_term: 1 leader_uuid: "d29e6306ecb4424a8b10a1d655482bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29e6306ecb4424a8b10a1d655482bc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40133 } } }
I20260927 21:06:08.909235 6113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:08.909291 6098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:08.911240 6113 catalog_manager.cc:1347] Generated new cluster ID: 076d67c34c4d4f78b191cf3f6f16a2da
I20260927 21:06:08.911340 6113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:08.911528 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:08.906581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:44401 (local address 127.0.0.1:40133)
0927 21:06:08.906623 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:08.906625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:08.906932 (+ 307us) server_negotiation.cc:407] Connection header received
0927 21:06:08.907054 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:08.907056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:08.907069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:08.907139 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:08.907955 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:08.908927 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:08.909964 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:08.910278 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:08.910371 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:08.911204 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:08.911205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:08.911206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:08.911211 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:08.911228 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:08.911233 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:08.911297 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:08.911443 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 21:06:08.911456 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:06:08.957260 6113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:08.957858 6113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:08.961570 6113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d29e6306ecb4424a8b10a1d655482bc9: Generated new TSK 0
I20260927 21:06:08.961761 6113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:08.965509 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6019
W20260927 21:06:08.970397 6004 connection.cc:535] client connection to 127.0.0.1:40133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40133): Connection reset by peer (error 104)
2026-09-27T21:06:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:09 dist-test-slave-6c7s krb5kdc[6119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:09 dist-test-slave-6c7s krb5kdc[6119](info): set up 2 sockets
Sep 27 21:06:09 dist-test-slave-6c7s krb5kdc[6119](info): commencing operation
krb5kdc: starting...
W20260927 21:06:10.006711 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:10 dist-test-slave-6c7s krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 21:06:10.140021 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:10.222222 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:10.222443 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:10.222468 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:10.223649 6132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:10.223706 6132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:10.223721 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:10.223733 6132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:10.223747 6132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:10.225373 6132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:10.225646 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:10.225899 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 21:06:10.231333 6132 server_base.cc:1034] running on GCE node
W20260927 21:06:10.231280 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:10.231348 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:10.231312 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:10.231669 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:10.232012 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:10.233336 6132 hybrid_clock.cc:648] HybridClock initialized: now 1790543170233300 us; error 58 us; skew 500 ppm
Sep 27 21:06:10 dist-test-slave-6c7s krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:10.236024 6132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 21:06:10.236593 6132 webserver.cc:460] Webserver started at http://127.0.0.1:39071/ using document root <none> and password file <none>
I20260927 21:06:10.236912 6132 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:10.236981 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:10.237149 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:10.238251 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "66bffd6403b74ee2a50afb38fc25f789"
format_stamp: "Formatted at 2026-09-27 21:06:10 on dist-test-slave-6c7s"
server_key: "b7859afbc82208a9f4b7fe37ad06d418"
server_key_iv: "d858b64de971417e90a9edfec3e3f1af"
server_key_version: "encryptionkey@0"
I20260927 21:06:10.238626 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "66bffd6403b74ee2a50afb38fc25f789"
format_stamp: "Formatted at 2026-09-27 21:06:10 on dist-test-slave-6c7s"
server_key: "b7859afbc82208a9f4b7fe37ad06d418"
server_key_iv: "d858b64de971417e90a9edfec3e3f1af"
server_key_version: "encryptionkey@0"
I20260927 21:06:10.240335 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:10.241168 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:10.241361 6132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:10.241477 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "66bffd6403b74ee2a50afb38fc25f789"
format_stamp: "Formatted at 2026-09-27 21:06:10 on dist-test-slave-6c7s"
server_key: "b7859afbc82208a9f4b7fe37ad06d418"
server_key_iv: "d858b64de971417e90a9edfec3e3f1af"
server_key_version: "encryptionkey@0"
I20260927 21:06:10.241562 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:10.244966 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:10.247215 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:10.247382 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:10.251268 6132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44843
I20260927 21:06:10.251317 6200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44843 every 8 connection(s)
I20260927 21:06:10.251642 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:10.252661 6201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:10.255033 6201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789: Bootstrap starting.
I20260927 21:06:10.255120 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6132
I20260927 21:06:10.255319 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:10.255541 17147 external_mini_cluster.cc:1428] Setting key 9dafb0d1e2082283de9dd41d872cfe32
I20260927 21:06:10.255738 6201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:10.256065 6201 log.cc:824] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:10.256872 6201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789: No bootstrap required, opened a new log
I20260927 21:06:10.258158 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:10.256070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:56325 (local address 127.0.0.1:44843)
0927 21:06:10.256319 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:10.256326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:10.256352 (+ 26us) server_negotiation.cc:407] Connection header received
0927 21:06:10.256848 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:10.256857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:10.256892 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:10.257056 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:10.257696 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:10.257700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:10.257700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:10.257704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:10.257741 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:10.257752 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:10.257822 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:10.257906 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 21:06:10.257968 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260927 21:06:10.259176 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:10.259303 6201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 21:06:10.259482 6201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:10.259512 6201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bffd6403b74ee2a50afb38fc25f789, State: Initialized, Role: FOLLOWER
I20260927 21:06:10.259632 6201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 21:06:10.259727 6201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:10.259757 6201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:10.259809 6201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:10.260658 6201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 21:06:10.260756 6201 leader_election.cc:302] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bffd6403b74ee2a50afb38fc25f789; no voters:
I20260927 21:06:10.260967 6201 leader_election.cc:288] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:10.261014 6210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:10.261176 6210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [term 1 LEADER]: Becoming Leader. State: Replica: 66bffd6403b74ee2a50afb38fc25f789, State: Running, Role: LEADER
I20260927 21:06:10.261289 6210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } }
I20260927 21:06:10.261308 6201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:10.261691 6213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bffd6403b74ee2a50afb38fc25f789. Latest consensus state: current_term: 1 leader_uuid: "66bffd6403b74ee2a50afb38fc25f789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260927 21:06:10.261932 6213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:10.261852 6212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bffd6403b74ee2a50afb38fc25f789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bffd6403b74ee2a50afb38fc25f789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44843 } } }
I20260927 21:06:10.262169 6212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:10.262344 6215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:10.262871 6215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:10.263154 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:10.261001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.80:41667 (local address 127.0.0.1:44843)
0927 21:06:10.261044 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:10.261046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:10.261485 (+ 439us) server_negotiation.cc:407] Connection header received
0927 21:06:10.261797 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:10.261800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:10.261810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:10.261903 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:10.262760 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:10.262761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:10.262762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:10.262766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:10.262781 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:10.262786 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:10.262837 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:10.263019 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:06:10.263036 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:06:10.264708 6215 catalog_manager.cc:1347] Generated new cluster ID: c3f0b2f668cb4406b110510cdd41a0b0
I20260927 21:06:10.264767 6215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:10.277412 6215 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:10.277613 6215 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:10.282056 6215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bffd6403b74ee2a50afb38fc25f789: Generated new TSK 0
I20260927 21:06:10.282264 6215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:10.285482 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6132
2026-09-27T21:06:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:10 dist-test-slave-6c7s krb5kdc[6232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:10 dist-test-slave-6c7s krb5kdc[6232](info): set up 2 sockets
Sep 27 21:06:10 dist-test-slave-6c7s krb5kdc[6232](info): commencing operation
krb5kdc: starting...
W20260927 21:06:11.337872 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:11.465740 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40297
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:11.549683 6245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:11.549921 6245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:11.549947 6245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:11.551134 6245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:11.551184 6245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:11.551199 6245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:11.551213 6245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:11.551225 6245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:11.552737 6245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:40297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:11.553001 6245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:11.553189 6245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:11.558244 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:11.558292 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:11.558342 6253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:11.558614 6245 server_base.cc:1034] running on GCE node
I20260927 21:06:11.558943 6245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:11.559222 6245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:11.560425 6245 hybrid_clock.cc:648] HybridClock initialized: now 1790543171560420 us; error 37 us; skew 500 ppm
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543171, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:11.563231 6245 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:11.563731 6245 webserver.cc:460] Webserver started at http://127.16.190.254:46703/ using document root <none> and password file <none>
I20260927 21:06:11.563982 6245 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:11.564062 6245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:11.564213 6245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:11.565363 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "344c59ee26e84339911b081dc10ceb23"
format_stamp: "Formatted at 2026-09-27 21:06:11 on dist-test-slave-6c7s"
server_key: "93920ca552ae07c79c64d50e4ab42e1f"
server_key_iv: "c9ae37feddc9db9c5df904a156650b0d"
server_key_version: "encryptionkey@0"
I20260927 21:06:11.565743 6245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "344c59ee26e84339911b081dc10ceb23"
format_stamp: "Formatted at 2026-09-27 21:06:11 on dist-test-slave-6c7s"
server_key: "93920ca552ae07c79c64d50e4ab42e1f"
server_key_iv: "c9ae37feddc9db9c5df904a156650b0d"
server_key_version: "encryptionkey@0"
I20260927 21:06:11.567195 6245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 21:06:11.568104 6260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:11.568317 6245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:11.568428 6245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "344c59ee26e84339911b081dc10ceb23"
format_stamp: "Formatted at 2026-09-27 21:06:11 on dist-test-slave-6c7s"
server_key: "93920ca552ae07c79c64d50e4ab42e1f"
server_key_iv: "c9ae37feddc9db9c5df904a156650b0d"
server_key_version: "encryptionkey@0"
I20260927 21:06:11.568507 6245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:11.578397 6245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:11.580876 6245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:11.581092 6245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:11.585580 6245 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:40297
I20260927 21:06:11.585639 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:40297 every 8 connection(s)
I20260927 21:06:11.586052 6245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:11.586913 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:11.589293 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23: Bootstrap starting.
I20260927 21:06:11.589859 6313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:11.590104 6313 log.cc:824] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:11.590816 6313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23: No bootstrap required, opened a new log
I20260927 21:06:11.591037 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6245
I20260927 21:06:11.591171 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:11.591372 17147 external_mini_cluster.cc:1428] Setting key b9b8268f78842dedb64eff24609e0435
I20260927 21:06:11.593616 6313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } }
I20260927 21:06:11.593806 6313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:11.593847 6313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344c59ee26e84339911b081dc10ceb23, State: Initialized, Role: FOLLOWER
I20260927 21:06:11.594010 6313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } }
I20260927 21:06:11.594079 6313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:11.594101 6313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:11.594172 6313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:11.594825 6313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } }
I20260927 21:06:11.594933 6313 leader_election.cc:302] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 344c59ee26e84339911b081dc10ceb23; no voters:
I20260927 21:06:11.595088 6313 leader_election.cc:288] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:11.595216 6318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:11.595351 6313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:11.595376 6318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [term 1 LEADER]: Becoming Leader. State: Replica: 344c59ee26e84339911b081dc10ceb23, State: Running, Role: LEADER
I20260927 21:06:11.595587 6318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } }
I20260927 21:06:11.596016 6320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "344c59ee26e84339911b081dc10ceb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } } }
I20260927 21:06:11.596096 6320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:11.596235 6320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 344c59ee26e84339911b081dc10ceb23. Latest consensus state: current_term: 1 leader_uuid: "344c59ee26e84339911b081dc10ceb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344c59ee26e84339911b081dc10ceb23" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 40297 } } }
I20260927 21:06:11.596283 6320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:11.596503 6327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:11.597318 6327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:11.600523 6327 catalog_manager.cc:1347] Generated new cluster ID: 06ca44ac4879479fbc11a3f3d7df603b
I20260927 21:06:11.600586 6327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:11.605582 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:11.591916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57081 (local address 127.16.190.254:40297)
0927 21:06:11.592127 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:11.592138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:11.592175 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:06:11.593561 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:11.593571 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:11.593640 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:11.593800 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:11.594636 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:11.597267 (+ 2631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:11.598676 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:11.599307 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:11.599446 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:11.602144 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:11.602149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:11.602149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:11.602161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:11.603847 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:11.604314 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:11.604317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:11.604318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:11.604417 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:11.604646 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:11.604647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:11.604647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:11.604960 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:11.605148 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:11.605250 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 21:06:11.605353 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260927 21:06:11.606621 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:11.615110 6327 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:11.615437 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:11.608000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49757 (local address 127.16.190.254:40297)
0927 21:06:11.608038 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:11.608040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:11.608187 (+ 147us) server_negotiation.cc:407] Connection header received
0927 21:06:11.608253 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:11.608254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:11.608264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:11.608313 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:11.609157 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:11.609708 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:11.610452 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:11.610712 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:11.610791 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:11.612063 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:11.612065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:11.612065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:11.612073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:11.613745 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:11.614296 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:11.614298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:11.614298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:11.614378 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:11.614619 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:11.614620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:11.614621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:11.614845 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:11.614977 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:11.615178 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 21:06:11.615369 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 21:06:11.616142 6327 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:11.622257 6327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 344c59ee26e84339911b081dc10ceb23: Generated new TSK 0
I20260927 21:06:11.622539 6327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:11.629169 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6245
W20260927 21:06:11.633972 6230 connection.cc:535] client connection to 127.16.190.254:40297 recv error: Network error: failed to read from TLS socket (remote: 127.16.190.254:40297): Connection reset by peer (error 104)
2026-09-27T21:06:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6344](info): set up 2 sockets
Sep 27 21:06:11 dist-test-slave-6c7s krb5kdc[6344](info): commencing operation
krb5kdc: starting...
W20260927 21:06:12.725617 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:12 dist-test-slave-6c7s krb5kdc[6344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:12.853312 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36397
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:36397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:12.936285 6357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:12.936546 6357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:12.936585 6357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:12.937791 6357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:12.937863 6357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:12.937886 6357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:12.937907 6357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:12.937935 6357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:12.939419 6357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:36397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:36397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:12.939693 6357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:12.939917 6357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:12.944904 6362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:12.944905 6365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:12.944931 6363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:12.945276 6357 server_base.cc:1034] running on GCE node
I20260927 21:06:12.945542 6357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:12.945874 6357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:12.947257 6357 hybrid_clock.cc:648] HybridClock initialized: now 1790543172947247 us; error 38 us; skew 500 ppm
Sep 27 21:06:12 dist-test-slave-6c7s krb5kdc[6344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543172, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:12.956353 6357 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:12.956936 6357 webserver.cc:460] Webserver started at http://127.16.190.254:42117/ using document root <none> and password file <none>
I20260927 21:06:12.957185 6357 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:12.957259 6357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:12.957405 6357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:12.958575 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "69ce8d64fc16430fb4750f890da51096"
format_stamp: "Formatted at 2026-09-27 21:06:12 on dist-test-slave-6c7s"
server_key: "9ee93c90550f88b0bdd5f8b4a7f7001d"
server_key_iv: "b76cebbb2fb9545990d30cb40e4daccf"
server_key_version: "encryptionkey@0"
I20260927 21:06:12.958940 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "69ce8d64fc16430fb4750f890da51096"
format_stamp: "Formatted at 2026-09-27 21:06:12 on dist-test-slave-6c7s"
server_key: "9ee93c90550f88b0bdd5f8b4a7f7001d"
server_key_iv: "b76cebbb2fb9545990d30cb40e4daccf"
server_key_version: "encryptionkey@0"
I20260927 21:06:12.960368 6357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:12.961174 6372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:12.961402 6357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:06:12.961522 6357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "69ce8d64fc16430fb4750f890da51096"
format_stamp: "Formatted at 2026-09-27 21:06:12 on dist-test-slave-6c7s"
server_key: "9ee93c90550f88b0bdd5f8b4a7f7001d"
server_key_iv: "b76cebbb2fb9545990d30cb40e4daccf"
server_key_version: "encryptionkey@0"
I20260927 21:06:12.961607 6357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:12.974756 6357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:12.977273 6357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:12.977447 6357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:12.982139 6357 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:36397
I20260927 21:06:12.982175 6424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:36397 every 8 connection(s)
I20260927 21:06:12.982585 6357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:12.983556 6425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:12.986027 6425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096: Bootstrap starting.
I20260927 21:06:12.986665 6425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:12.986956 6425 log.cc:824] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:12.987756 6425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096: No bootstrap required, opened a new log
I20260927 21:06:12.988632 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6357
I20260927 21:06:12.988766 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:12.989007 17147 external_mini_cluster.cc:1428] Setting key b4c316ba7f25a29a97ffd29e8ddd2a37
I20260927 21:06:12.989909 6425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } }
I20260927 21:06:12.990085 6425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:12.990128 6425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ce8d64fc16430fb4750f890da51096, State: Initialized, Role: FOLLOWER
I20260927 21:06:12.990342 6425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } }
I20260927 21:06:12.990456 6425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:12.990489 6425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:12.990536 6425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:12.991310 6425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } }
I20260927 21:06:12.991417 6425 leader_election.cc:302] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ce8d64fc16430fb4750f890da51096; no voters:
I20260927 21:06:12.991567 6425 leader_election.cc:288] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:12.991694 6430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:12.991802 6425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:12.991846 6430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [term 1 LEADER]: Becoming Leader. State: Replica: 69ce8d64fc16430fb4750f890da51096, State: Running, Role: LEADER
I20260927 21:06:12.991976 6430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } }
I20260927 21:06:12.992364 6432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ce8d64fc16430fb4750f890da51096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } } }
I20260927 21:06:12.992475 6432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:12.992669 6432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ce8d64fc16430fb4750f890da51096. Latest consensus state: current_term: 1 leader_uuid: "69ce8d64fc16430fb4750f890da51096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ce8d64fc16430fb4750f890da51096" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 36397 } } }
I20260927 21:06:12.992746 6439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:12.992781 6432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:12.993340 6439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:12.995237 6439 catalog_manager.cc:1347] Generated new cluster ID: 8494c8bc1d35422e848dcd9f1e2c55e9
I20260927 21:06:12.995296 6439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:12.995704 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:12.989624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42695 (local address 127.16.190.254:36397)
0927 21:06:12.989825 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:12.989835 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:12.989869 (+ 34us) server_negotiation.cc:407] Connection header received
0927 21:06:12.989985 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:12.989992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:12.990047 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:12.990196 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:12.990458 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:12.992226 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:12.993507 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:12.994104 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:12.994208 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:12.994988 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:12.994992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:12.994992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:12.995218 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:12.995252 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:12.995263 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:12.995359 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:12.995497 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 21:06:12.995526 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260927 21:06:12.996680 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:13.008921 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:13.005247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51639 (local address 127.16.190.254:36397)
0927 21:06:13.005305 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:13.005308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:13.005546 (+ 238us) server_negotiation.cc:407] Connection header received
0927 21:06:13.005642 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:13.005643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:13.005656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:13.005716 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:13.005899 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:13.006447 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:13.007272 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:13.007525 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:13.007605 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:13.008569 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:13.008570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:13.008570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:13.008576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:13.008592 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:13.008596 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:13.008653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:13.008758 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 21:06:13.008774 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 21:06:13.016206 6439 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:13.017023 6439 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:13.020303 6439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ce8d64fc16430fb4750f890da51096: Generated new TSK 0
I20260927 21:06:13.020510 6439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:13.026259 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6357
W20260927 21:06:13.030400 6342 connection.cc:535] client connection to 127.16.190.254:36397 recv error: Network error: failed to read from TLS socket (remote: 127.16.190.254:36397): Connection reset by peer (error 104)
2026-09-27T21:06:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:13 dist-test-slave-6c7s krb5kdc[6456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:13 dist-test-slave-6c7s krb5kdc[6456](info): set up 2 sockets
Sep 27 21:06:13 dist-test-slave-6c7s krb5kdc[6456](info): commencing operation
krb5kdc: starting...
W20260927 21:06:14.089800 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:14 dist-test-slave-6c7s krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:14.216537 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38069
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:38069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:14.299440 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:14.299728 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:14.299772 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:14.301106 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:14.301182 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:14.301208 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:14.301239 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:14.301265 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:14.302842 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:38069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:14.303141 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:14.303378 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:14.308446 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:14.308475 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:14.308472 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:14.308751 6469 server_base.cc:1034] running on GCE node
I20260927 21:06:14.309115 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:14.309432 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:14.310654 6469 hybrid_clock.cc:648] HybridClock initialized: now 1790543174310611 us; error 41 us; skew 500 ppm
Sep 27 21:06:14 dist-test-slave-6c7s krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543174, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:14.313460 6469 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:14.313979 6469 webserver.cc:460] Webserver started at http://127.16.190.254:46849/ using document root <none> and password file <none>
I20260927 21:06:14.314218 6469 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:14.314267 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:14.314360 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:14.315431 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "febf087232ee464d96fb52573e87ab5b"
format_stamp: "Formatted at 2026-09-27 21:06:14 on dist-test-slave-6c7s"
server_key: "858b5e2e1a5fa57ec5e91352d23be18e"
server_key_iv: "7b0598a167c93f6e0b47ff512419af6c"
server_key_version: "encryptionkey@0"
I20260927 21:06:14.315762 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "febf087232ee464d96fb52573e87ab5b"
format_stamp: "Formatted at 2026-09-27 21:06:14 on dist-test-slave-6c7s"
server_key: "858b5e2e1a5fa57ec5e91352d23be18e"
server_key_iv: "7b0598a167c93f6e0b47ff512419af6c"
server_key_version: "encryptionkey@0"
I20260927 21:06:14.317296 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:06:14.318079 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:14.318291 6469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:06:14.318390 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "febf087232ee464d96fb52573e87ab5b"
format_stamp: "Formatted at 2026-09-27 21:06:14 on dist-test-slave-6c7s"
server_key: "858b5e2e1a5fa57ec5e91352d23be18e"
server_key_iv: "7b0598a167c93f6e0b47ff512419af6c"
server_key_version: "encryptionkey@0"
I20260927 21:06:14.318460 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:14.322022 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:14.324265 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:14.324373 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:14.328197 6469 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:38069
I20260927 21:06:14.328249 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:38069 every 8 connection(s)
I20260927 21:06:14.328557 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:14.329591 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:14.331820 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6469
I20260927 21:06:14.331897 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b: Bootstrap starting.
I20260927 21:06:14.331954 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:14.332134 17147 external_mini_cluster.cc:1428] Setting key afa1740430758f54efc33978f811cba4
I20260927 21:06:14.332561 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:14.332970 6537 log.cc:824] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:14.333752 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b: No bootstrap required, opened a new log
I20260927 21:06:14.334606 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:14.332637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43143 (local address 127.16.190.254:38069)
0927 21:06:14.332839 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:14.332848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:14.332884 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:06:14.333352 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:14.333360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:14.333386 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:14.333500 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:14.334165 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:14.334168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:14.334168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:14.334172 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:14.334207 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:14.334216 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:14.334291 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:14.334361 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 21:06:14.334409 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260927 21:06:14.335752 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:14.336318 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } }
I20260927 21:06:14.336503 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:14.336540 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febf087232ee464d96fb52573e87ab5b, State: Initialized, Role: FOLLOWER
I20260927 21:06:14.336674 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } }
I20260927 21:06:14.336768 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:14.336799 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:14.336871 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:14.337661 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } }
I20260927 21:06:14.337785 6537 leader_election.cc:302] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: febf087232ee464d96fb52573e87ab5b; no voters:
I20260927 21:06:14.337947 6537 leader_election.cc:288] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:14.337975 6547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:14.338191 6547 raft_consensus.cc:686] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [term 1 LEADER]: Becoming Leader. State: Replica: febf087232ee464d96fb52573e87ab5b, State: Running, Role: LEADER
I20260927 21:06:14.338213 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:14.338339 6547 consensus_queue.cc:237] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } }
I20260927 21:06:14.338512 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:14.337409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34975 (local address 127.16.190.254:38069)
0927 21:06:14.337447 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:14.337449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:14.337523 (+ 74us) server_negotiation.cc:407] Connection header received
0927 21:06:14.337593 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:14.337594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:14.337597 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:14.337643 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:14.338260 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:14.338261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:14.338261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:14.338266 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:14.338281 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:14.338286 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:14.338317 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:14.338418 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 21:06:14.338430 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:06:14.339622 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader febf087232ee464d96fb52573e87ab5b. Latest consensus state: current_term: 1 leader_uuid: "febf087232ee464d96fb52573e87ab5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } } }
I20260927 21:06:14.339701 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:14.339622 6548 sys_catalog.cc:455] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "febf087232ee464d96fb52573e87ab5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febf087232ee464d96fb52573e87ab5b" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 38069 } } }
I20260927 21:06:14.339747 6548 sys_catalog.cc:458] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:14.339884 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:14.340361 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:14.341996 6562 catalog_manager.cc:1347] Generated new cluster ID: 3ff6d1d005a04167a54bd7c880e599e3
I20260927 21:06:14.342047 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:14.353626 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:14.353826 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:14.360395 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P febf087232ee464d96fb52573e87ab5b: Generated new TSK 0
I20260927 21:06:14.360620 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:14.368525 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6469
2026-09-27T21:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:14 dist-test-slave-6c7s krb5kdc[6568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:14 dist-test-slave-6c7s krb5kdc[6568](info): set up 2 sockets
Sep 27 21:06:14 dist-test-slave-6c7s krb5kdc[6568](info): commencing operation
krb5kdc: starting...
W20260927 21:06:15.437850 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:15.563617 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:44887
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:44887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:15.645470 6581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:15.645738 6581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:15.645778 6581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:15.646935 6581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:15.647010 6581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:15.647035 6581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:15.647064 6581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:15.647094 6581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:15.648571 6581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:44887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:15.648862 6581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:15.649091 6581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:15.654006 6589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:15.654026 6586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:15.654008 6587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:15.654366 6581 server_base.cc:1034] running on GCE node
I20260927 21:06:15.654593 6581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:15.654922 6581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:15.656102 6581 hybrid_clock.cc:648] HybridClock initialized: now 1790543175656092 us; error 35 us; skew 500 ppm
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543175, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:15.658680 6581 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:15.659159 6581 webserver.cc:460] Webserver started at http://127.16.190.254:45727/ using document root <none> and password file <none>
I20260927 21:06:15.659359 6581 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:15.659401 6581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:15.659488 6581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:15.660507 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "bdf682062287480f849156d857ad74c5"
format_stamp: "Formatted at 2026-09-27 21:06:15 on dist-test-slave-6c7s"
server_key: "9f7a16e912aa2499ef2a5d844fe3c08f"
server_key_iv: "4a8b9811e82dea37c2f47788210eaf74"
server_key_version: "encryptionkey@0"
I20260927 21:06:15.660882 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "bdf682062287480f849156d857ad74c5"
format_stamp: "Formatted at 2026-09-27 21:06:15 on dist-test-slave-6c7s"
server_key: "9f7a16e912aa2499ef2a5d844fe3c08f"
server_key_iv: "4a8b9811e82dea37c2f47788210eaf74"
server_key_version: "encryptionkey@0"
I20260927 21:06:15.662251 6581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:06:15.663009 6596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:15.663187 6581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:06:15.663280 6581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "bdf682062287480f849156d857ad74c5"
format_stamp: "Formatted at 2026-09-27 21:06:15 on dist-test-slave-6c7s"
server_key: "9f7a16e912aa2499ef2a5d844fe3c08f"
server_key_iv: "4a8b9811e82dea37c2f47788210eaf74"
server_key_version: "encryptionkey@0"
I20260927 21:06:15.663347 6581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:15.680107 6581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:15.682417 6581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:15.682541 6581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:15.686496 6581 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:44887
I20260927 21:06:15.686585 6648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:44887 every 8 connection(s)
I20260927 21:06:15.686947 6581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:15.687889 6649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:15.688550 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6581
I20260927 21:06:15.688674 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:15.688930 17147 external_mini_cluster.cc:1428] Setting key b5503cc338800eb3c50077ae65c9eaa5
I20260927 21:06:15.691039 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5: Bootstrap starting.
I20260927 21:06:15.691778 6649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:15.692093 6649 log.cc:824] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:15.692886 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5: No bootstrap required, opened a new log
I20260927 21:06:15.695188 6649 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } }
I20260927 21:06:15.695338 6649 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:15.695362 6649 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf682062287480f849156d857ad74c5, State: Initialized, Role: FOLLOWER
I20260927 21:06:15.695508 6649 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } }
I20260927 21:06:15.695582 6649 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:15.695605 6649 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:15.695680 6649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:15.696399 6649 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } }
I20260927 21:06:15.696506 6649 leader_election.cc:302] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf682062287480f849156d857ad74c5; no voters:
I20260927 21:06:15.696683 6649 leader_election.cc:288] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:15.696712 6654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:15.696863 6654 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [term 1 LEADER]: Becoming Leader. State: Replica: bdf682062287480f849156d857ad74c5, State: Running, Role: LEADER
I20260927 21:06:15.696972 6649 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:15.697002 6654 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } }
I20260927 21:06:15.697695 6655 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf682062287480f849156d857ad74c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } } }
I20260927 21:06:15.697788 6655 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:15.698168 6664 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf682062287480f849156d857ad74c5. Latest consensus state: current_term: 1 leader_uuid: "bdf682062287480f849156d857ad74c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf682062287480f849156d857ad74c5" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 44887 } } }
I20260927 21:06:15.698318 6664 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:15.698539 6668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:15.699147 6668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:15.700683 6668 catalog_manager.cc:1347] Generated new cluster ID: a91eeb076a604fa3a5136f9fc00ea541
I20260927 21:06:15.700736 6668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:15.701229 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:15.689514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52979 (local address 127.16.190.254:44887)
0927 21:06:15.689712 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:15.689723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:15.689758 (+ 35us) server_negotiation.cc:407] Connection header received
0927 21:06:15.690307 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:15.690316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:15.690382 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:15.690538 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:15.691146 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:15.692475 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:15.693333 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:15.693969 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:15.694093 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:15.696965 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:15.696969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:15.696969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:15.696981 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:15.699199 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:15.699627 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:15.699630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:15.699631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:15.699743 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:15.699970 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:15.699971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:15.699971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:15.700273 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:15.700433 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:15.700906 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 21:06:15.701040 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260927 21:06:15.702102 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:15.712212 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:15.703529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48663 (local address 127.16.190.254:44887)
0927 21:06:15.703561 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:15.703562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:15.705093 (+ 1531us) server_negotiation.cc:407] Connection header received
0927 21:06:15.705125 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:15.705126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:15.705137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:15.705222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:15.705822 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:15.706573 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:15.707552 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:15.707877 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:15.707947 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:15.709347 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:15.709348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:15.709348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:15.709355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:15.710964 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:15.711337 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:15.711339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:15.711339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:15.711399 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:15.711578 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:15.711579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:15.711579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:15.711720 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:15.711817 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:15.711929 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 21:06:15.712008 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:06:15.721331 6668 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:15.722095 6668 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:15.733767 6668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf682062287480f849156d857ad74c5: Generated new TSK 0
I20260927 21:06:15.734004 6668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:15.736632 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6581
2026-09-27T21:06:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6681](info): set up 2 sockets
Sep 27 21:06:15 dist-test-slave-6c7s krb5kdc[6681](info): commencing operation
krb5kdc: starting...
W20260927 21:06:16.785940 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:16 dist-test-slave-6c7s krb5kdc[6681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:16.913553 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39903
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:38925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:17.001629 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:17.001933 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:17.001976 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:17.003672 6694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:17.003953 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:17.004019 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:17.004046 6694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:17.004076 6694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:17.006093 6694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:17.006449 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:17.006712 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:17.012209 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:17.012209 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:17.012442 6694 server_base.cc:1034] running on GCE node
W20260927 21:06:17.012209 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:17.012755 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:17.013103 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:17.014328 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790543177014312 us; error 32 us; skew 500 ppm
Sep 27 21:06:17 dist-test-slave-6c7s krb5kdc[6681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:17.017278 6694 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:17.017814 6694 webserver.cc:460] Webserver started at http://127.16.190.254:43039/ using document root <none> and password file <none>
I20260927 21:06:17.018041 6694 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:17.018087 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:17.018201 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:17.019397 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "39e93d317f7d4de889250c7534d9c011"
format_stamp: "Formatted at 2026-09-27 21:06:17 on dist-test-slave-6c7s"
server_key: "78216d768ae6da3ec5406550f77ae4c8"
server_key_iv: "5523e96df66329ed5639d6c2dffecefe"
server_key_version: "encryptionkey@0"
I20260927 21:06:17.019728 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "39e93d317f7d4de889250c7534d9c011"
format_stamp: "Formatted at 2026-09-27 21:06:17 on dist-test-slave-6c7s"
server_key: "78216d768ae6da3ec5406550f77ae4c8"
server_key_iv: "5523e96df66329ed5639d6c2dffecefe"
server_key_version: "encryptionkey@0"
I20260927 21:06:17.021270 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:17.022063 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:17.022351 6694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:17.022552 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "39e93d317f7d4de889250c7534d9c011"
format_stamp: "Formatted at 2026-09-27 21:06:17 on dist-test-slave-6c7s"
server_key: "78216d768ae6da3ec5406550f77ae4c8"
server_key_iv: "5523e96df66329ed5639d6c2dffecefe"
server_key_version: "encryptionkey@0"
I20260927 21:06:17.022634 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:17.032274 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:17.034945 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:17.035128 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:17.039892 6694 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39903
I20260927 21:06:17.039932 6761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39903 every 8 connection(s)
I20260927 21:06:17.040323 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:17.041347 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:17.044013 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011: Bootstrap starting.
I20260927 21:06:17.044711 6762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:17.045090 6762 log.cc:824] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:17.045961 6762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011: No bootstrap required, opened a new log
I20260927 21:06:17.047935 6762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } }
I20260927 21:06:17.048107 6762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:17.048133 6762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39e93d317f7d4de889250c7534d9c011, State: Initialized, Role: FOLLOWER
I20260927 21:06:17.048309 6762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } }
I20260927 21:06:17.048380 6762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:17.048424 6762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:17.048482 6762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:17.048975 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6694
I20260927 21:06:17.049108 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:17.049299 17147 external_mini_cluster.cc:1428] Setting key 520b475ca0ccf014ef6a4f7add50cee2
I20260927 21:06:17.049893 6762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } }
I20260927 21:06:17.050037 6762 leader_election.cc:302] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39e93d317f7d4de889250c7534d9c011; no voters:
I20260927 21:06:17.050275 6762 leader_election.cc:288] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:17.050311 6767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:17.050432 6767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [term 1 LEADER]: Becoming Leader. State: Replica: 39e93d317f7d4de889250c7534d9c011, State: Running, Role: LEADER
I20260927 21:06:17.050526 6767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } }
I20260927 21:06:17.050599 6762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:17.050894 6768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39e93d317f7d4de889250c7534d9c011. Latest consensus state: current_term: 1 leader_uuid: "39e93d317f7d4de889250c7534d9c011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } } }
I20260927 21:06:17.051002 6768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:17.051162 6769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39e93d317f7d4de889250c7534d9c011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39e93d317f7d4de889250c7534d9c011" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39903 } } }
I20260927 21:06:17.051221 6769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:17.051324 6771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:17.051961 6771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:17.053714 6771 catalog_manager.cc:1347] Generated new cluster ID: 57a3ccd596514b24a418f3a6ec62098b
I20260927 21:06:17.053777 6771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:17.056172 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:17.049855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59971 (local address 127.16.190.254:39903)
0927 21:06:17.050093 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:17.050104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:17.050140 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:06:17.050263 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:17.050270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:17.050392 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:17.050783 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:17.050900 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:17.052603 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:17.053774 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:17.054487 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:17.054632 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:17.055388 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:17.055392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:17.055393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:17.055632 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:17.055674 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:17.055692 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:17.055777 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:17.055855 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 21:06:17.055930 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260927 21:06:17.057300 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:17.062666 6766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:17.058842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42919 (local address 127.16.190.254:39903)
0927 21:06:17.058883 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:17.058885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:17.059043 (+ 158us) server_negotiation.cc:407] Connection header received
0927 21:06:17.059096 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:17.059097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:17.059120 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:17.059190 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:17.059334 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:17.060181 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:17.061191 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:17.061541 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:17.061617 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:17.062381 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:17.062383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:17.062383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:17.062391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:17.062409 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:17.062415 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:17.062482 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:17.062535 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 21:06:17.062567 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:06:17.069049 6771 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:17.069875 6771 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:17.080591 6771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39e93d317f7d4de889250c7534d9c011: Generated new TSK 0
I20260927 21:06:17.080888 6771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:17.084125 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6694
2026-09-27T21:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:17 dist-test-slave-6c7s krb5kdc[6793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:17 dist-test-slave-6c7s krb5kdc[6793](info): set up 2 sockets
Sep 27 21:06:17 dist-test-slave-6c7s krb5kdc[6793](info): commencing operation
krb5kdc: starting...
W20260927 21:06:18.162026 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:18 dist-test-slave-6c7s krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:18.289304 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39009
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:39009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:18.372197 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:18.372480 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:18.372519 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:18.373782 6806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:18.373869 6806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:18.373893 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:18.373914 6806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:18.373945 6806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:18.375521 6806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:39009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:39009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:18.375826 6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:18.376063 6806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:18.381212 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:18.381227 6806 server_base.cc:1034] running on GCE node
W20260927 21:06:18.381191 6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:18.381192 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:18.381639 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:18.381960 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:18.383142 6806 hybrid_clock.cc:648] HybridClock initialized: now 1790543178383124 us; error 44 us; skew 500 ppm
Sep 27 21:06:18 dist-test-slave-6c7s krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543178, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:18.390388 6806 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:18.390900 6806 webserver.cc:460] Webserver started at http://127.16.190.254:37691/ using document root <none> and password file <none>
I20260927 21:06:18.391151 6806 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:18.391222 6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:18.391367 6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:18.392473 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "af2191ece6984d1c978ac0c179b210d3"
format_stamp: "Formatted at 2026-09-27 21:06:18 on dist-test-slave-6c7s"
server_key: "3d18c587bdfb0c6c2d04b6560f7b2e30"
server_key_iv: "c2decc25ec9431960abbbbfceaa0b643"
server_key_version: "encryptionkey@0"
I20260927 21:06:18.392900 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "af2191ece6984d1c978ac0c179b210d3"
format_stamp: "Formatted at 2026-09-27 21:06:18 on dist-test-slave-6c7s"
server_key: "3d18c587bdfb0c6c2d04b6560f7b2e30"
server_key_iv: "c2decc25ec9431960abbbbfceaa0b643"
server_key_version: "encryptionkey@0"
I20260927 21:06:18.394291 6806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:18.395085 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:18.395478 6806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:18.395571 6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "af2191ece6984d1c978ac0c179b210d3"
format_stamp: "Formatted at 2026-09-27 21:06:18 on dist-test-slave-6c7s"
server_key: "3d18c587bdfb0c6c2d04b6560f7b2e30"
server_key_iv: "c2decc25ec9431960abbbbfceaa0b643"
server_key_version: "encryptionkey@0"
I20260927 21:06:18.395653 6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:18.399173 6806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:18.401316 6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:18.401466 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:18.405424 6806 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:39009
I20260927 21:06:18.405457 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:39009 every 8 connection(s)
I20260927 21:06:18.405804 6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:18.406745 6874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:18.409212 6874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3: Bootstrap starting.
I20260927 21:06:18.409785 6874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:18.410063 6874 log.cc:824] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:18.410781 6874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3: No bootstrap required, opened a new log
I20260927 21:06:18.412387 6874 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } }
I20260927 21:06:18.412525 6874 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 21:06:18.412549 6874 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2191ece6984d1c978ac0c179b210d3, State: Initialized, Role: FOLLOWER
I20260927 21:06:18.412704 6874 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } }
I20260927 21:06:18.412767 6874 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 21:06:18.412788 6874 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 21:06:18.412866 6874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:18.413694 6874 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } }
I20260927 21:06:18.413785 6874 leader_election.cc:302] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2191ece6984d1c978ac0c179b210d3; no voters:
I20260927 21:06:18.413995 6874 leader_election.cc:288] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 21:06:18.414098 6877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:18.414244 6877 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [term 1 LEADER]: Becoming Leader. State: Replica: af2191ece6984d1c978ac0c179b210d3, State: Running, Role: LEADER
I20260927 21:06:18.414256 6874 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:18.414378 6877 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } }
I20260927 21:06:18.414472 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6806
I20260927 21:06:18.414595 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:18.414708 6877 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2191ece6984d1c978ac0c179b210d3. Latest consensus state: current_term: 1 leader_uuid: "af2191ece6984d1c978ac0c179b210d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } } }
I20260927 21:06:18.414813 17147 external_mini_cluster.cc:1428] Setting key 1732efad97d12646072e9c7c2551041a
I20260927 21:06:18.414710 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2191ece6984d1c978ac0c179b210d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2191ece6984d1c978ac0c179b210d3" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 39009 } } }
I20260927 21:06:18.414913 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:18.415118 6883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:18.415791 6883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:18.415127 6877 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:18.418031 6883 catalog_manager.cc:1347] Generated new cluster ID: bf6199d786234cbb9e17b4cbe2ce89fd
I20260927 21:06:18.418085 6883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:18.418416 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:18.415284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53109 (local address 127.16.190.254:39009)
0927 21:06:18.416424 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:18.416433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:18.416469 (+ 36us) server_negotiation.cc:407] Connection header received
0927 21:06:18.416748 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:18.416756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:18.416790 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:18.416994 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:18.417880 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:18.417884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:18.417884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:18.417889 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:18.417932 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:18.417947 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:18.418188 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:18.418225 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 21:06:18.418269 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":79,"threads_started":1}
I20260927 21:06:18.419251 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:18.421955 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:18.420663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47597 (local address 127.16.190.254:39009)
0927 21:06:18.420720 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:18.420722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:18.420886 (+ 164us) server_negotiation.cc:407] Connection header received
0927 21:06:18.421002 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:18.421003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:18.421009 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:18.421036 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:18.421723 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:18.421724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:18.421724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 21:06:18.421728 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:18.421743 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 21:06:18.421747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:18.421794 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:18.421846 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:06:18.421862 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:06:18.432803 6883 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:18.433027 6883 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:18.435618 6883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2191ece6984d1c978ac0c179b210d3: Generated new TSK 0
I20260927 21:06:18.435791 6883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:18.445323 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6806
2026-09-27T21:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1361 ms)
[----------] 18 tests from AuthTokenIssuingTest (24433 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:18 dist-test-slave-6c7s krb5kdc[6905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:18 dist-test-slave-6c7s krb5kdc[6905](info): set up 2 sockets
Sep 27 21:06:18 dist-test-slave-6c7s krb5kdc[6905](info): commencing operation
krb5kdc: starting...
W20260927 21:06:19.502115 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:19.633070 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42181
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:19.715134 6918 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:19.715372 6918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:19.715390 6918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:19.715412 6918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:19.716557 6918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:19.716594 6918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:19.716609 6918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:19.716620 6918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:19.716632 6918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:19.718097 6918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:19.718305 6918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:19.718477 6918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:19.723462 6924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:19.723472 6918 server_base.cc:1034] running on GCE node
W20260927 21:06:19.723574 6923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:19.723474 6926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:19.723949 6918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:19.724253 6918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:19.725419 6918 hybrid_clock.cc:648] HybridClock initialized: now 1790543179725405 us; error 27 us; skew 500 ppm
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:19.727953 6918 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:19.728399 6918 webserver.cc:460] Webserver started at http://127.16.190.254:38787/ using document root <none> and password file <none>
I20260927 21:06:19.728624 6918 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:19.728693 6918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:19.728868 6918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:19.729987 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "7b7423b2bfc74f7baa0df03c3f223f39"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "9f51a82ed6a2e98d3e28b247328381ac"
server_key_iv: "d7437a609c63b434fa9a4f8bf4b95559"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.730369 6918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "7b7423b2bfc74f7baa0df03c3f223f39"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "9f51a82ed6a2e98d3e28b247328381ac"
server_key_iv: "d7437a609c63b434fa9a4f8bf4b95559"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.731757 6918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 21:06:19.732568 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:19.732791 6918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:19.732931 6918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "7b7423b2bfc74f7baa0df03c3f223f39"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "9f51a82ed6a2e98d3e28b247328381ac"
server_key_iv: "d7437a609c63b434fa9a4f8bf4b95559"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.733012 6918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:19.755051 6918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:19.757241 6918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:19.757416 6918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:19.761328 6918 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:42181
I20260927 21:06:19.761377 6985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:42181 every 8 connection(s)
I20260927 21:06:19.761741 6918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:19.762653 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:19.764307 6986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:19.768383 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6918
I20260927 21:06:19.768509 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:19.768707 17147 external_mini_cluster.cc:1428] Setting key b57b8204fc88c3a71402986d18a9ab86
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:19.775362 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.765080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42181 (local address 127.0.0.1:55236)
0927 21:06:19.765243 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.765295 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:19.765806 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:19.766153 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:19.766156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:19.766449 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:19.766609 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.766613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.767691 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.767692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.768574 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.768577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.769167 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.769168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.769258 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.769829 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:19.769840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:19.771811 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:19.773683 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.773691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.773693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.773997 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.774225 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.774226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.774227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.774348 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.774707 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:19.774714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:19.774802 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 21:06:19.774907 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 21:06:19.775084 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260927 21:06:19.775400 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.765101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.16.190.254:42181)
0927 21:06:19.765295 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:19.765306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:19.765396 (+ 90us) server_negotiation.cc:407] Connection header received
0927 21:06:19.765987 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:19.765989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:19.766033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:19.766143 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:19.766738 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.767605 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.768699 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.769082 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.769166 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.771909 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:19.771916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:19.771916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:19.771922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:19.773604 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.774085 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.774087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.774087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.774162 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.774421 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.774422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.774422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.774637 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:19.774701 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:19.775172 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 21:06:19.775271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260927 21:06:19.776535 6986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.190.253@KRBTEST.COM; defaulting to no policy
I20260927 21:06:19.776978 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.776787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51560)
0927 21:06:19.776888 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.776942 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 21:06:19.777133 6935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
I20260927 21:06:19.777575 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.777492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51562)
0927 21:06:19.777533 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.777557 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 21:06:19.777678 6986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:37527: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
I20260927 21:06:19.799968 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:37527
--webserver_interface=127.16.190.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:06:19.826923 6986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } attempt: 1
I20260927 21:06:19.827517 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.827363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51568)
0927 21:06:19.827429 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.827480 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:06:19.828011 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.827897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51580)
0927 21:06:19.827944 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.827980 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:06:19.828128 6986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:37527: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
W20260927 21:06:19.883507 6993 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:19.883757 6993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:19.883776 6993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:19.883798 6993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:19.885056 6993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:19.885102 6993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:19.885116 6993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:19.885135 6993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:19.885191 6993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:19.886694 6993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.190.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:19.886955 6993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:19.887189 6993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:19.892107 6999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:19.892082 6998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:19.892125 6993 server_base.cc:1034] running on GCE node
W20260927 21:06:19.892082 7001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:19.892603 6993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:19.892931 6993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:19.894089 6993 hybrid_clock.cc:648] HybridClock initialized: now 1790543179894056 us; error 47 us; skew 500 ppm
I20260927 21:06:19.896283 6986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } attempt: 2
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:19.896783 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.896642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51594)
0927 21:06:19.896712 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.896742 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 21:06:19.897013 6993 init.cc:377] Logged in from keytab as kudu/127.16.190.253@KRBTEST.COM (short username kudu)
I20260927 21:06:19.897195 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.897062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51598)
0927 21:06:19.897109 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.897161 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:06:19.897292 6986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:37527: Network error: Client connection negotiation failed: client connection to 127.16.190.253:37527: connect: Connection refused (error 111)
I20260927 21:06:19.897553 6993 webserver.cc:460] Webserver started at http://127.16.190.253:43339/ using document root <none> and password file <none>
I20260927 21:06:19.897814 6993 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:19.897861 6993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:19.898013 6993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:19.899150 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data/instance:
uuid: "4c5c51943d924684851470c1d70ef93e"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "c583abcb26d3d0c7904de32df246c07e"
server_key_iv: "55131b1983d4ce63429cdb3518033457"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.899537 6993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal/instance:
uuid: "4c5c51943d924684851470c1d70ef93e"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "c583abcb26d3d0c7904de32df246c07e"
server_key_iv: "55131b1983d4ce63429cdb3518033457"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.901069 6993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 21:06:19.901940 7008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:19.902185 6993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:19.902288 6993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "4c5c51943d924684851470c1d70ef93e"
format_stamp: "Formatted at 2026-09-27 21:06:19 on dist-test-slave-6c7s"
server_key: "c583abcb26d3d0c7904de32df246c07e"
server_key_iv: "55131b1983d4ce63429cdb3518033457"
server_key_version: "encryptionkey@0"
I20260927 21:06:19.902371 6993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:19.926244 6993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:19.928442 6993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:19.928649 6993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:19.932875 6993 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.253:37527
I20260927 21:06:19.932914 7060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.253:37527 every 8 connection(s)
I20260927 21:06:19.933290 6993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
I20260927 21:06:19.934180 7061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:19.935567 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 6993
I20260927 21:06:19.935672 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-1/wal/instance
I20260927 21:06:19.936012 17147 external_mini_cluster.cc:1428] Setting key efa981e10cf9faedba67c907d86cea54
I20260927 21:06:19.936018 7061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.252@KRBTEST.COM; defaulting to no policy
I20260927 21:06:19.946329 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.936871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.16.190.254:42181)
0927 21:06:19.936927 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:19.936930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:19.937182 (+ 252us) server_negotiation.cc:407] Connection header received
0927 21:06:19.937726 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:19.937727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:19.937742 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:19.937807 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:19.938470 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.938981 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.940002 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.940232 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.940313 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.942860 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:19.942862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:19.942862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:19.942869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:19.944641 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.945263 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.945264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.945264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.945331 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.945648 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.945649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.945649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.945777 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:19.945841 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:19.946145 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 21:06:19.946225 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:06:19.946468 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.936855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42181 (local address 127.0.0.1:55238)
0927 21:06:19.937028 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.937069 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:19.937613 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:19.937883 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:19.937890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:19.938162 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:19.938375 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.938379 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.939086 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.939087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.939896 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.939898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.940334 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.940335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.940464 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.941072 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:19.941081 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:19.942732 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:19.944803 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.944858 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.944862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.945147 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.945425 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.945427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.945427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.945558 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.945847 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:19.945854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:19.945996 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 21:06:19.946093 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 21:06:19.946250 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.252@KRBTEST.COM" created.
I20260927 21:06:19.947148 7061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:19.956650 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.948457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51600)
0927 21:06:19.948520 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.948532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:19.948574 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:19.948903 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:19.948904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:19.949057 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:19.949109 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.949111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.950017 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.950018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.950614 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.950616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.951115 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.951116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.951198 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.951724 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:19.951728 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:19.953260 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:19.955389 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.955391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.955391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.955561 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.955840 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.955841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.955842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.955925 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.956311 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:19.956313 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:19.956354 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 21:06:19.956423 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 21:06:19.956575 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:06:19.956800 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.948545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.16.190.253:37527)
0927 21:06:19.948713 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:19.948717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:19.948738 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:06:19.948762 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:19.948763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:19.948774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:19.948901 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:19.949190 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.949917 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.950702 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.951024 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.951114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.953358 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:19.953367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:19.953367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:19.953376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:19.955296 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.955641 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.955644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.955644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.955734 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.956039 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.956040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.956041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.956204 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:19.956289 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:19.956485 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 21:06:19.956656 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260927 21:06:19.957543 7061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:19.957975 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.957812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55206)
0927 21:06:19.957892 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.957949 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 21:06:19.958210 7011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
I20260927 21:06:19.958662 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.958546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55216)
0927 21:06:19.958601 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.958630 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 21:06:19.958797 7061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:36385: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
I20260927 21:06:19.965011 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:36385
--webserver_interface=127.16.190.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:06:19.988482 6986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } attempt: 3
Sep 27 21:06:19 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:19.996541 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.988949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51606)
0927 21:06:19.989022 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.989037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:19.989109 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:19.989285 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:19.989287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:19.989463 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:19.989533 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.989534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.990496 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.990497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.991052 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:19.991054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.991578 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.991579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:19.991641 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.992229 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:19.992235 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:19.993609 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:19.995496 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.995497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.995497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.995666 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.995899 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:19.995900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:19.995901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:19.995964 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:19.996271 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:19.996272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:19.996316 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 21:06:19.996384 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 21:06:19.996467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:06:19.996599 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.989004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.16.190.253:37527)
0927 21:06:19.989085 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:19.989087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:19.989100 (+ 13us) server_negotiation.cc:407] Connection header received
0927 21:06:19.989162 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:19.989163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:19.989177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:19.989242 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:19.989625 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.990396 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.991150 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:19.991491 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:19.991564 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:19.993706 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:19.993707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:19.993708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:19.993716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:19.995408 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.995747 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.995748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.995748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.995812 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:19.996037 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:19.996037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:19.996038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:19.996176 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:19.996251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:19.996429 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 21:06:19.996510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 21:06:19.997448 6986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:19.997972 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.997857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55226)
0927 21:06:19.997911 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.997947 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 21:06:19.998351 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:19.998219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55236)
0927 21:06:19.998299 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:19.998331 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 21:06:19.998468 6986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:36385: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
I20260927 21:06:20.008049 7061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } attempt: 1
I20260927 21:06:20.008548 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.008413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55252)
0927 21:06:20.008477 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.008518 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 21:06:20.009024 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.008923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55260)
0927 21:06:20.008969 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.008995 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 21:06:20.009166 7061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:36385: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
I20260927 21:06:20.029630 6986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } attempt: 1
I20260927 21:06:20.030155 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.029998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55270)
0927 21:06:20.030068 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.030106 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:06:20.030544 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.030446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55274)
0927 21:06:20.030494 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.030521 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 21:06:20.030648 6986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:36385: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
W20260927 21:06:20.058945 7068 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:20.059264 7068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:20.059297 7068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:20.059350 7068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:20.060839 7068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:20.060921 7068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:20.060948 7068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:20.060981 7068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:20.061017 7068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:20.062716 7068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:42181,127.16.190.253:37527,127.16.190.252:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.190.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:20.063040 7068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:20.063318 7068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:20.068706 7074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:20.068712 7076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:20.068706 7073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:20.069020 7068 server_base.cc:1034] running on GCE node
I20260927 21:06:20.069278 7068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:20.069592 7068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:20.070745 7068 hybrid_clock.cc:648] HybridClock initialized: now 1790543180070732 us; error 39 us; skew 500 ppm
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:20.073545 7068 init.cc:377] Logged in from keytab as kudu/127.16.190.252@KRBTEST.COM (short username kudu)
I20260927 21:06:20.074098 7068 webserver.cc:460] Webserver started at http://127.16.190.252:40437/ using document root <none> and password file <none>
I20260927 21:06:20.074364 7068 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:20.074445 7068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:20.074608 7068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:20.075820 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data/instance:
uuid: "f81aa1db31074afabcb39845563cf377"
format_stamp: "Formatted at 2026-09-27 21:06:20 on dist-test-slave-6c7s"
server_key: "c37a017f932ae6438256de7fb5260e74"
server_key_iv: "cac3a3b4b2fbc53d1bdeb6a46e04da7b"
server_key_version: "encryptionkey@0"
I20260927 21:06:20.076236 7068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal/instance:
uuid: "f81aa1db31074afabcb39845563cf377"
format_stamp: "Formatted at 2026-09-27 21:06:20 on dist-test-slave-6c7s"
server_key: "c37a017f932ae6438256de7fb5260e74"
server_key_iv: "cac3a3b4b2fbc53d1bdeb6a46e04da7b"
server_key_version: "encryptionkey@0"
I20260927 21:06:20.077322 7061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } attempt: 2
I20260927 21:06:20.077724 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.077596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55286)
0927 21:06:20.077659 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.077694 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 21:06:20.077947 7068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260927 21:06:20.078135 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.078002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55302)
0927 21:06:20.078079 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.078112 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 21:06:20.078293 7061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:36385: Network error: Client connection negotiation failed: client connection to 127.16.190.252:36385: connect: Connection refused (error 111)
I20260927 21:06:20.078881 7083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:20.079068 7068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:06:20.079145 7068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "f81aa1db31074afabcb39845563cf377"
format_stamp: "Formatted at 2026-09-27 21:06:20 on dist-test-slave-6c7s"
server_key: "c37a017f932ae6438256de7fb5260e74"
server_key_iv: "cac3a3b4b2fbc53d1bdeb6a46e04da7b"
server_key_version: "encryptionkey@0"
I20260927 21:06:20.079241 7068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:20.091522 7068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:20.093607 7068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:20.093775 7068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:20.097834 7068 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.252:36385
I20260927 21:06:20.098012 7135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.252:36385 every 8 connection(s)
I20260927 21:06:20.098290 7068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
I20260927 21:06:20.099211 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:20.100380 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 7068
I20260927 21:06:20.100467 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790542906842326-17147-0/minicluster-data/master-2/wal/instance
I20260927 21:06:20.100678 17147 external_mini_cluster.cc:1428] Setting key e9502b55b900cc69a87cf4559f0c245e
I20260927 21:06:20.101042 7136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:20.105816 6986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } attempt: 2
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:06:20.111670 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.101889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:42181 (local address 127.0.0.1:55252)
0927 21:06:20.102041 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.102086 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:20.102199 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:20.102408 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:20.102411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:20.102677 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:20.102844 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.102849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.103569 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.103571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.104377 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.104379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.104873 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.104874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.105039 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.105640 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:20.105648 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:20.107509 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:20.109789 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.109795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.109797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.110074 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.110604 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.110606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.110606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.110750 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.111075 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:20.111081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:20.111171 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 21:06:20.111277 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 21:06:20.111441 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260927 21:06:20.111838 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.101283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.16.190.252:36385)
0927 21:06:20.101473 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.101483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.101515 (+ 32us) server_negotiation.cc:407] Connection header received
0927 21:06:20.102132 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.102142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.102207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.102306 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.102948 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.103848 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.104693 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.105217 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.105311 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.108174 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.108182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.108182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.108189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.110219 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.110587 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.110589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.110590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.110677 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.110895 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.110896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.110897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.111178 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.111397 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.111622 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.111746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260927 21:06:20.111976 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.101936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.16.190.254:42181)
0927 21:06:20.102001 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.102003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.102173 (+ 170us) server_negotiation.cc:407] Connection header received
0927 21:06:20.102295 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.102297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.102307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.102375 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.102945 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.103485 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.104458 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.104761 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.104859 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.107595 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.107596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.107596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.107603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.109687 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.110163 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.110165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.110165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.110232 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.110839 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.110840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.110840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.110973 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.111007 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.111315 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.111397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 21:06:20.112624 7136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:20.114439 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.106076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55326)
0927 21:06:20.106254 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.106266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:20.106314 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:20.106497 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:20.106498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:20.106650 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:20.106680 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.106681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.107422 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.107423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.108136 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.108139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.108620 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.108621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.108683 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.109192 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:20.109197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:20.110583 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:20.113320 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.113321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.113321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.113545 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.113765 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.113766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.113766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.113834 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.114127 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:20.114128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:20.114174 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:06:20.114253 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 21:06:20.114348 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 21:06:20.114527 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.106116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.16.190.252:36385)
0927 21:06:20.106306 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.106308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.106317 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:06:20.106375 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.106376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.106387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.106487 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.106767 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.107329 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.108221 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.108526 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.108800 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.110666 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.110667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.110667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.110673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.113233 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.113626 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.113627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.113627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.113689 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.113910 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.113911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.113912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.114049 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.114127 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.114309 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.114392 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:20.118014 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: Bootstrap starting.
I20260927 21:06:20.118640 6986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: Neither blocks nor log segments found. Creating new log.
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:20.118993 6986 log.cc:824] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:20.119922 6986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: No bootstrap required, opened a new log
I20260927 21:06:20.120570 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.112961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.16.190.253:37527)
0927 21:06:20.113013 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.113015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.113096 (+ 81us) server_negotiation.cc:407] Connection header received
0927 21:06:20.113163 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.113164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.113177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.113231 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.113482 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.114064 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.115026 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.115341 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.115412 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.117639 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.117640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.117641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.117648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.119375 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.119775 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.119776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.119776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.119837 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.120035 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.120036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.120037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.120155 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.120245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.120428 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.120506 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 21:06:20.120589 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.112903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:37527 (local address 127.0.0.1:51610)
0927 21:06:20.113041 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.113052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:20.113106 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:20.113233 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:20.113234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:20.113376 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:20.113417 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.113418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.114165 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.114166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.114927 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.114930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.115427 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.115428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.115505 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.116239 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:20.116244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:20.117548 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:20.119459 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.119461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.119461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.119683 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.119920 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.119921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.119921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.119979 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.120237 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:20.120238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:20.120284 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 21:06:20.120394 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 21:06:20.120488 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 21:06:20.121541 7136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:20.122328 6986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.122478 6986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7423b2bfc74f7baa0df03c3f223f39, State: Initialized, Role: FOLLOWER
I20260927 21:06:20.122602 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.113284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.16.190.254:42181)
0927 21:06:20.113318 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.113320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.113571 (+ 251us) server_negotiation.cc:407] Connection header received
0927 21:06:20.113669 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.113670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.113679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.113722 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.114390 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.114916 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.115813 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.116085 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.116151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.119326 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.119327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.119328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.119334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.121325 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.121757 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.121759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.121759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.121815 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.122014 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.122016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.122016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.122166 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.122317 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.122417 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.122500 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:06:20.122557 6986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.122934 6986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:20.124311 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.124399 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:06:20.125010 7154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:20.125077 7154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543180, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:20.126080 6918 master_runner.cc:186] Error getting master registration for 127.16.190.253:37527: OK, instance_id { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" instance_seqno: 1790543179931570 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:06:20.129480 6918 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:06:20.130229 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.121829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55332)
0927 21:06:20.121872 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.121883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:20.121918 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:20.122092 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:20.122093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:20.122204 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:20.122259 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.122261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.123397 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.123398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.124203 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.124205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.124892 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.124893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.124956 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.125561 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:20.125566 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:20.127052 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:20.128957 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.128958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.128958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.129171 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.129385 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.129386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.129386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.129440 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.129822 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:20.129823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:20.129849 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 21:06:20.129929 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 21:06:20.130062 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":3}
I20260927 21:06:20.130318 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.121871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.16.190.252:36385)
0927 21:06:20.121912 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.121913 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.121921 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:06:20.121965 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.121966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.121976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.122026 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.122337 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.122980 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.124286 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.124551 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.124598 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.127164 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.127166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.127166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.127172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.128796 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.129254 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.129255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.129256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.129321 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.129553 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.129554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.129554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.129750 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.129788 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.129954 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.130143 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":22}
I20260927 21:06:20.132268 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.123186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.16.190.253:37527)
0927 21:06:20.123230 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.123231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.123240 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:06:20.123276 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.123277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.123286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.123339 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.123921 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.124602 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.125628 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.125919 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.125979 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.129243 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.129245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.129245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.129254 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.130956 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.131385 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.131387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.131388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.131454 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.131690 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.131691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.131691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.131850 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.132038 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.132086 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.132175 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 21:06:20.132407 7136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: Bootstrap starting.
I20260927 21:06:20.133385 7136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:20.133805 7136 log.cc:824] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:20.134157 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:20.134748 7136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: No bootstrap required, opened a new log
I20260927 21:06:20.135424 6960 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7b7423b2bfc74f7baa0df03c3f223f39"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55266
I20260927 21:06:20.135627 6960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:06:20.135684 6960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:20.136634 6960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.136936 6960 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5c51943d924684851470c1d70ef93e (127.16.190.253:37527), f81aa1db31074afabcb39845563cf377 (127.16.190.252:36385)
I20260927 21:06:20.137182 7136 raft_consensus.cc:348] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.137355 7136 raft_consensus.cc:729] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81aa1db31074afabcb39845563cf377, State: Initialized, Role: FOLLOWER
I20260927 21:06:20.137486 7136 consensus_queue.cc:260] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.137871 7136 sys_catalog.cc:564] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:20.137884 7157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.138144 7110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f81aa1db31074afabcb39845563cf377"
I20260927 21:06:20.138262 7110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:20.138176 7157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.139642 7110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7423b2bfc74f7baa0df03c3f223f39 in term 1.
I20260927 21:06:20.139946 6936 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7423b2bfc74f7baa0df03c3f223f39, f81aa1db31074afabcb39845563cf377; no voters:
I20260927 21:06:20.140095 7143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:20.140278 7143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7423b2bfc74f7baa0df03c3f223f39, State: Running, Role: LEADER
I20260927 21:06:20.140383 7143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
W20260927 21:06:20.140699 7172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:20.140754 7172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:20.140892 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7423b2bfc74f7baa0df03c3f223f39. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.140972 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:20.142654 7068 master_runner.cc:186] Error getting master registration for 127.16.190.253:37527: OK, instance_id { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" instance_seqno: 1790543179931570 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:06:20.145064 7068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:06:20.153328 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.145223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.16.190.253:37527)
0927 21:06:20.145275 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.145277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.145345 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:06:20.145453 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.145454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.145465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.145494 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.146397 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.147163 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.148149 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.148455 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.148500 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.150216 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.150218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.150218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.150225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.151948 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.152343 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.152344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.152344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.152402 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.152631 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.152632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.152633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.152783 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.152882 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.153115 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.153194 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 21:06:20.155675 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.145413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.16.190.254:42181)
0927 21:06:20.145463 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.145466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.145808 (+ 342us) server_negotiation.cc:407] Connection header received
0927 21:06:20.145957 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.145958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.145984 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.146017 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.147090 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.147818 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.149327 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.149655 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.149702 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.152195 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.152196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.152197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.152205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.154481 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.154874 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.154875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.154876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.154930 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.155157 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.155158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.155158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.155280 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.155390 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.155499 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.155571 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:06:20.156955 7176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:20.157217 7176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:20.158722 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.145669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.16.190.252:36385)
0927 21:06:20.145717 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.145719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.148954 (+ 3235us) server_negotiation.cc:407] Connection header received
0927 21:06:20.149059 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.149060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.149070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.149100 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.149964 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.150774 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.151995 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.152343 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.152400 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.155894 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.155896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.155896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.155904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.157569 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.157951 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.157952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.157952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.158021 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.158216 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.158217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.158217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.158326 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.158444 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.158550 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.158626 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 21:06:20.159034 7110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [term 1 FOLLOWER]: Refusing update from remote peer 7b7423b2bfc74f7baa0df03c3f223f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:06:20.159368 7143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:06:20.161062 7157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7423b2bfc74f7baa0df03c3f223f39. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.161147 7157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.166340 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.166446 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:20.167423 7157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.167516 7157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.167941 7176 catalog_manager.cc:1347] Generated new cluster ID: 07bb828550004a62a20909d3c70509b6
I20260927 21:06:20.167995 7176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:20.168012 7181 mvcc.cc:205] Tried to move back new op lower bound from 7334064865928585216 to 7334064865855913984. Current Snapshot: MvccSnapshot[applied={T|T < 7334064865928585216}]
I20260927 21:06:20.169405 7061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } attempt: 3
Sep 27 21:06:20 dist-test-slave-6c7s krb5kdc[6905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543179, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:06:20.177805 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.169661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:36385 (local address 127.0.0.1:55350)
0927 21:06:20.169787 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:20.169799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:20.169836 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:20.170069 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:20.170070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:20.170261 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:20.170306 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.170308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.171219 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.171220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.172003 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:20.172004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.172505 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.172506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:20.172570 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.173063 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:20.173069 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:20.174582 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:20.176667 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.176668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.176669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.176893 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.177136 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:20.177137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:20.177137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:20.177213 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:20.177497 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:20.177498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:20.177552 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 21:06:20.177608 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 21:06:20.177705 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 21:06:20.178406 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:20.169899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.16.190.252:36385)
0927 21:06:20.169933 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.169934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.169940 (+ 6us) server_negotiation.cc:407] Connection header received
0927 21:06:20.169961 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.169961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.169970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.170010 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.170385 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.171118 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.172095 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:20.172392 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:20.172440 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:20.174931 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:20.174932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:20.174932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:20.174940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:20.176594 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.176976 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.176978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.176978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.177042 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:20.177282 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:20.177282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:20.177283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:20.177415 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:20.177462 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:20.178254 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 21:06:20.178322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 21:06:20.180203 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: Bootstrap starting.
I20260927 21:06:20.180917 7061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:20.181201 7061 log.cc:824] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:20.185647 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: No bootstrap required, opened a new log
I20260927 21:06:20.188401 7061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.188633 7061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5c51943d924684851470c1d70ef93e, State: Initialized, Role: FOLLOWER
I20260927 21:06:20.188846 7061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } }
I20260927 21:06:20.189515 7034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:20.190414 7034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [term 1 FOLLOWER]: Refusing update from remote peer 7b7423b2bfc74f7baa0df03c3f223f39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:06:20.189538 7035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c5c51943d924684851470c1d70ef93e"
I20260927 21:06:20.190811 7184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:06:20.191345 7035 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7423b2bfc74f7baa0df03c3f223f39 in term 1.
I20260927 21:06:20.189437 7061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:20.196983 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.197227 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.197319 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7423b2bfc74f7baa0df03c3f223f39. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.197415 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.197476 7196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.197557 7196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:20.198074 7191 mvcc.cc:205] Tried to move back new op lower bound from 7334064865928585216 to 7334064865855913984. Current Snapshot: MvccSnapshot[applied={T|T < 7334064865928585216}]
W20260927 21:06:20.198814 7202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:20.199013 7202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:20.199041 7184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7423b2bfc74f7baa0df03c3f223f39" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 42181 } } peers { permanent_uuid: "4c5c51943d924684851470c1d70ef93e" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 37527 } } peers { permanent_uuid: "f81aa1db31074afabcb39845563cf377" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 36385 } } }
I20260927 21:06:20.199146 7184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39 [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:20.199976 7176 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:20.200654 7176 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:20.206661 7176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7423b2bfc74f7baa0df03c3f223f39: Generated new TSK 0
I20260927 21:06:20.206877 7176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 21:06:20.234797 6989 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.233143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.16.190.254:42181)
0927 21:06:20.233195 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.233197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.234127 (+ 930us) server_negotiation.cc:407] Connection header received
0927 21:06:20.234236 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.234237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.234250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.234280 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.234724 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55292: BlockingRecv error: recv got EOF from 127.0.0.1:55292 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 21:06:20.235589 7141 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.233522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.16.190.252:36385)
0927 21:06:20.233562 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.233564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.235072 (+ 1508us) server_negotiation.cc:407] Connection header received
0927 21:06:20.235169 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.235170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.235181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.235211 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.235529 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55364: BlockingRecv error: recv got EOF from 127.0.0.1:55364 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:06:20.236436 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.233854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.16.190.253:37527)
0927 21:06:20.233894 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.233896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.235799 (+ 1903us) server_negotiation.cc:407] Connection header received
0927 21:06:20.235976 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.235977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.235987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.236017 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.236376 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51624: BlockingRecv error: recv got EOF from 127.0.0.1:51624 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:06:20.238936 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.237780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.16.190.253:37527)
0927 21:06:20.237814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.237816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.238381 (+ 565us) server_negotiation.cc:407] Connection header received
0927 21:06:20.238491 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.238492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.238510 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.238539 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.238891 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51630: BlockingRecv error: recv got EOF from 127.0.0.1:51630 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 21:06:20.242121 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.241298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.16.190.253:37527)
0927 21:06:20.241333 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.241335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.241582 (+ 247us) server_negotiation.cc:407] Connection header received
0927 21:06:20.241684 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.241685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.241710 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.241741 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.242074 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:06:20.246186 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.245390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.190.253:37527)
0927 21:06:20.245427 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.245429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.245638 (+ 209us) server_negotiation.cc:407] Connection header received
0927 21:06:20.245752 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.245753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.245764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.245793 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.246132 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51652: BlockingRecv error: recv got EOF from 127.0.0.1:51652 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:06:20.252439 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.251502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.16.190.253:37527)
0927 21:06:20.251546 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.251548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.251830 (+ 282us) server_negotiation.cc:407] Connection header received
0927 21:06:20.251940 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.251941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.251953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.251984 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.252380 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51656: BlockingRecv error: recv got EOF from 127.0.0.1:51656 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 21:06:20.262899 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.261914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.16.190.253:37527)
0927 21:06:20.261959 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.261961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.262245 (+ 284us) server_negotiation.cc:407] Connection header received
0927 21:06:20.262356 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.262358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.262371 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.262402 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.262819 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51672: BlockingRecv error: recv got EOF from 127.0.0.1:51672 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:06:20.281456 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.280439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.16.190.253:37527)
0927 21:06:20.280514 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.280516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.280839 (+ 323us) server_negotiation.cc:407] Connection header received
0927 21:06:20.280952 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.280953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.280967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.280996 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.281395 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51674: BlockingRecv error: recv got EOF from 127.0.0.1:51674 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 21:06:20.316372 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.315254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.16.190.253:37527)
0927 21:06:20.315309 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.315312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.315649 (+ 337us) server_negotiation.cc:407] Connection header received
0927 21:06:20.315770 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.315771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.315785 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.315817 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.316305 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51690: BlockingRecv error: recv got EOF from 127.0.0.1:51690 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 21:06:20.382853 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.382040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.16.190.253:37527)
0927 21:06:20.382102 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.382104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.382219 (+ 115us) server_negotiation.cc:407] Connection header received
0927 21:06:20.382311 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.382312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.382327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.382365 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.382772 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51702: BlockingRecv error: recv got EOF from 127.0.0.1:51702 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 21:06:20.512780 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.512019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.16.190.253:37527)
0927 21:06:20.512139 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.512141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.512278 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:06:20.512311 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.512312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.512341 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.512409 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.512732 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv got EOF from 127.0.0.1:51704 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 21:06:20.770826 7066 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:20.770180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.16.190.253:37527)
0927 21:06:20.770298 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:20.770300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:20.770309 (+ 9us) server_negotiation.cc:407] Connection header received
0927 21:06:20.770344 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:20.770345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:20.770355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:20.770424 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:20.770772 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51718: BlockingRecv error: recv got EOF from 127.0.0.1:51718 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 21:06:21.141290 7172 catalog_manager.cc:1259] Loaded cluster ID: 07bb828550004a62a20909d3c70509b6
I20260927 21:06:21.141403 7172 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: loading cluster ID for follower catalog manager: success
I20260927 21:06:21.142252 7172 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: acquiring CA information for follower catalog manager: success
I20260927 21:06:21.142575 7172 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f81aa1db31074afabcb39845563cf377: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:06:21.199556 7202 catalog_manager.cc:1259] Loaded cluster ID: 07bb828550004a62a20909d3c70509b6
I20260927 21:06:21.199635 7202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: loading cluster ID for follower catalog manager: success
I20260927 21:06:21.200515 7202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: acquiring CA information for follower catalog manager: success
I20260927 21:06:21.200870 7202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c5c51943d924684851470c1d70ef93e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:06:21.287029 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:21.284176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.16.190.253:37527)
0927 21:06:21.284290 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:21.284292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:21.284378 (+ 86us) server_negotiation.cc:407] Connection header received
0927 21:06:21.284439 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:21.284440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:21.284489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:21.284554 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:21.284749 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.285346 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.286140 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.286379 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.286440 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:21.286579 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:21.286698 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:21.286773 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:21.286788 (+ 15us) server_negotiation.cc:299] Negotiation successful
0927 21:06:21.286808 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260927 21:06:21.291492 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:21.289043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.16.190.252:36385)
0927 21:06:21.289174 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:21.289176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:21.289201 (+ 25us) server_negotiation.cc:407] Connection header received
0927 21:06:21.289277 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:21.289278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:21.289316 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:21.289388 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:21.289491 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.290043 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.290783 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.291032 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.291089 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:21.291226 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:21.291333 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:21.291389 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:21.291407 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 21:06:21.291423 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260927 21:06:21.296206 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:21.293475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.16.190.252:36385)
0927 21:06:21.293510 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:21.293512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:21.293649 (+ 137us) server_negotiation.cc:407] Connection header received
0927 21:06:21.293682 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:21.293683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:21.293694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:21.293744 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:21.293938 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.294620 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295383 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.295658 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295726 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:21.295922 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:21.295955 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:21.296002 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:21.296101 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 21:06:21.296129 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 21:06:21.296259 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:21.293269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.16.190.253:37527)
0927 21:06:21.293311 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:21.293313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:21.293388 (+ 75us) server_negotiation.cc:407] Connection header received
0927 21:06:21.293447 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:21.293448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:21.293462 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:21.293520 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:21.293729 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.294502 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295489 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.295698 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295730 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:21.295986 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:21.296023 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:21.296050 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:21.296160 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 21:06:21.296184 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 21:06:21.296550 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:21.293194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.16.190.254:42181)
0927 21:06:21.293311 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:21.293314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:21.293487 (+ 173us) server_negotiation.cc:407] Connection header received
0927 21:06:21.293565 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:21.293566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:21.293623 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:21.293702 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:21.293902 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.294408 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295479 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:21.295795 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:21.295860 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:21.296044 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:21.296164 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:21.296224 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:21.296452 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 21:06:21.296477 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260927 21:06:21.298421 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6918
I20260927 21:06:21.303653 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 6993
I20260927 21:06:21.309099 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 7068
2026-09-27T21:06:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2865 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 21:06:21 dist-test-slave-6c7s krb5kdc[7287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 21:06:21 dist-test-slave-6c7s krb5kdc[7287](info): set up 2 sockets
Sep 27 21:06:21 dist-test-slave-6c7s krb5kdc[7287](info): commencing operation
krb5kdc: starting...
W20260927 21:06:22.358098 17147 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/test-user.keytab.
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T21:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T21:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.190.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
Entry for principal kudu/127.16.190.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab.
I20260927 21:06:22.488302 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:41161
--webserver_interface=127.16.190.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 21:06:22.571041 7300 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:22.571285 7300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:22.571305 7300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:22.571327 7300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:22.572471 7300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:22.572510 7300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:22.572525 7300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:22.572537 7300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:22.572549 7300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:22.574194 7300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.254.keytab
--principal=kudu/127.16.190.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.254:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.190.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:22.574460 7300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:22.574713 7300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:22.580137 7305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:22.580168 7300 server_base.cc:1034] running on GCE node
W20260927 21:06:22.580142 7308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:22.580142 7306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:22.580924 7300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:22.581313 7300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:22.582502 7300 hybrid_clock.cc:648] HybridClock initialized: now 1790543182582478 us; error 46 us; skew 500 ppm
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:22.585976 7300 init.cc:377] Logged in from keytab as kudu/127.16.190.254@KRBTEST.COM (short username kudu)
I20260927 21:06:22.586454 7300 webserver.cc:460] Webserver started at http://127.16.190.254:33813/ using document root <none> and password file <none>
I20260927 21:06:22.586700 7300 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:22.586769 7300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:22.586918 7300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:22.588295 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data/instance:
uuid: "bdcd4220b7644c2da588f255fe1181fa"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b1f318b7566ee929f048522b3e7e7d77"
server_key_iv: "f6ffe8526ae595a46b12a9a6bcf2fc3f"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.588757 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance:
uuid: "bdcd4220b7644c2da588f255fe1181fa"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b1f318b7566ee929f048522b3e7e7d77"
server_key_iv: "f6ffe8526ae595a46b12a9a6bcf2fc3f"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.590693 7300 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 21:06:22.591727 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:22.591985 7300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 21:06:22.592104 7300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal
uuid: "bdcd4220b7644c2da588f255fe1181fa"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b1f318b7566ee929f048522b3e7e7d77"
server_key_iv: "f6ffe8526ae595a46b12a9a6bcf2fc3f"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.592190 7300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:22.604911 7300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:22.606802 7300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:22.606916 7300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:22.610944 7300 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.254:41161
I20260927 21:06:22.610976 7367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.254:41161 every 8 connection(s)
I20260927 21:06:22.611367 7300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/data/info.pb
I20260927 21:06:22.612308 7368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:22.613323 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 7300
I20260927 21:06:22.613416 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-0/wal/instance
I20260927 21:06:22.613627 17147 external_mini_cluster.cc:1428] Setting key 9bd9329d7c44c303da6278011454575d
I20260927 21:06:22.614130 7368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.190.253@KRBTEST.COM; defaulting to no policy
I20260927 21:06:22.624258 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.614884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41161 (local address 127.0.0.1:39010)
0927 21:06:22.615061 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.615103 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.615579 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.615925 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.615927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.616182 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.616349 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.616354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.617471 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.617472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.618126 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.618128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.618619 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.618620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.618691 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.619233 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.619248 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.621037 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.622896 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.622902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.622903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.623098 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.623317 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.623318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.623318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.623396 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.623743 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.623749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.623821 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.623930 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.624147 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":80,"thread_start_us":82,"threads_started":1}
I20260927 21:06:22.624269 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.614944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.16.190.254:41161)
0927 21:06:22.615128 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.615135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.615203 (+ 68us) server_negotiation.cc:407] Connection header received
0927 21:06:22.615762 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.615766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.615795 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.615913 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.616437 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.617355 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.618228 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.618528 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.618640 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.621133 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.621141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.621141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.621147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.622811 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.623171 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.623174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.623174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.623254 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.623451 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.623452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.623452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.623660 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.623722 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.623968 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.624066 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260927 21:06:22.625085 7368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:22.625443 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.625355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60134)
0927 21:06:22.625386 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.625425 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 21:06:22.625586 7319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.253@KRBTEST.COM" created.
I20260927 21:06:22.626149 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.625931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60142)
0927 21:06:22.625978 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.626124 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:06:22.626256 7368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:34919: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
Entry for principal kudu/127.16.190.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab.
I20260927 21:06:22.644569 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:34919
--webserver_interface=127.16.190.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:06:22.675509 7368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } attempt: 1
I20260927 21:06:22.676070 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.675924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60148)
0927 21:06:22.675993 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.676040 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 21:06:22.676503 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.676365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60150)
0927 21:06:22.676424 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.676471 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:06:22.676668 7368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:34919: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
W20260927 21:06:22.730046 7375 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:22.730285 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:22.730304 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:22.730324 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:22.731542 7375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:22.731590 7375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:22.731606 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:22.731618 7375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:22.731630 7375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:22.733191 7375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.253.keytab
--principal=kudu/127.16.190.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.253:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.190.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:22.733470 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:22.733708 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:22.738857 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:22.738863 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:22.738862 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:22.739131 7375 server_base.cc:1034] running on GCE node
I20260927 21:06:22.739406 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:22.739727 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:22.740912 7375 hybrid_clock.cc:648] HybridClock initialized: now 1790543182740881 us; error 39 us; skew 500 ppm
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:22.743682 7375 init.cc:377] Logged in from keytab as kudu/127.16.190.253@KRBTEST.COM (short username kudu)
I20260927 21:06:22.744405 7375 webserver.cc:460] Webserver started at http://127.16.190.253:39703/ using document root <none> and password file <none>
I20260927 21:06:22.744766 7375 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:22.744884 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:22.744899 7368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } attempt: 2
I20260927 21:06:22.745046 7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:22.745280 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.745155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60160)
0927 21:06:22.745218 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.745246 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:06:22.745638 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.745530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60162)
0927 21:06:22.745582 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.745614 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 21:06:22.745735 7368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.253:34919: Network error: Client connection negotiation failed: client connection to 127.16.190.253:34919: connect: Connection refused (error 111)
I20260927 21:06:22.746374 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data/instance:
uuid: "5de926487e134f489861e9bd770ee8a4"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b3b26ef6b7657cf84c5d4838764c2375"
server_key_iv: "60c7b3917491590146007b5eb2bc64a5"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.746713 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal/instance:
uuid: "5de926487e134f489861e9bd770ee8a4"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b3b26ef6b7657cf84c5d4838764c2375"
server_key_iv: "60c7b3917491590146007b5eb2bc64a5"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.748495 7375 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 21:06:22.749485 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:22.749682 7375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 21:06:22.749789 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal
uuid: "5de926487e134f489861e9bd770ee8a4"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "b3b26ef6b7657cf84c5d4838764c2375"
server_key_iv: "60c7b3917491590146007b5eb2bc64a5"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.749878 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:22.757773 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:22.760125 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:22.760290 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:22.764142 7375 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.253:34919
I20260927 21:06:22.764181 7442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.253:34919 every 8 connection(s)
I20260927 21:06:22.764544 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/data/info.pb
I20260927 21:06:22.765548 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:22.767144 7443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:22.769901 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 7375
I20260927 21:06:22.770016 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-1/wal/instance
I20260927 21:06:22.770200 17147 external_mini_cluster.cc:1428] Setting key 999844dc9d4f56d2667762125c66095f
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:22.777940 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.768011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.16.190.254:41161)
0927 21:06:22.768072 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.768079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.768290 (+ 211us) server_negotiation.cc:407] Connection header received
0927 21:06:22.768932 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.768934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.768951 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.769024 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.769730 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.770545 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.771555 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.771764 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.771846 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.774578 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.774580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.774580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.774587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.776218 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.776745 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.776746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.776747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.776863 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.777212 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.777213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.777214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.777339 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.777397 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.777780 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.777860 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.16.190.252@KRBTEST.COM; defaulting to no policy
I20260927 21:06:22.778226 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.767947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41161 (local address 127.0.0.1:39024)
0927 21:06:22.768132 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.768176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.768702 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.769105 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.769109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.769377 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.769576 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.769581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.770640 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.770641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.771442 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.771444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.771852 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.771854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.771983 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.772570 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.772581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.774469 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.776313 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.776322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.776325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.776652 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.776955 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.776956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.776956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.777121 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.777415 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.777422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.777621 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.777755 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.777941 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260927 21:06:22.778826 7443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.190.252@KRBTEST.COM" created.
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:22.787293 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.779112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60170)
0927 21:06:22.779163 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.779177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.779230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.779519 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.779520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.779686 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.779745 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.779746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.780689 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.780690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.781258 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.781259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.781829 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.781830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.781901 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.782390 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.782404 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.783903 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.786115 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.786116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.786117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.786295 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.786525 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.786526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.786527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.786585 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.786897 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.786898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.786935 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.787021 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.787151 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 21:06:22.787366 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.779179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.16.190.253:34919)
0927 21:06:22.779354 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.779358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.779375 (+ 17us) server_negotiation.cc:407] Connection header received
0927 21:06:22.779422 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.779424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.779436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.779520 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.779836 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.780593 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.781354 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.781735 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.781831 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.784000 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.784011 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.784012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.784021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.785983 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.786395 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.786399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.786400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.786458 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.786649 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.786650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.786651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.786821 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.786911 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.787055 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.787210 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260927 21:06:22.788002 7443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:22.788405 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.788285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46778)
0927 21:06:22.788333 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.788380 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 21:06:22.788584 7391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
I20260927 21:06:22.789136 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.789045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46794)
0927 21:06:22.789090 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.789117 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 21:06:22.789275 7443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:40177: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
Entry for principal kudu/127.16.190.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab.
I20260927 21:06:22.801399 17147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQTa3tm/build/release/bin/kudu
/tmp/dist-test-taskQTa3tm/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:40177
--webserver_interface=127.16.190.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.190.212:38513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 21:06:22.836910 7368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } attempt: 3
I20260927 21:06:22.838541 7443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } attempt: 1
I20260927 21:06:22.838999 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.838815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46806)
0927 21:06:22.838896 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.838947 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 21:06:22.839387 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.839278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46820)
0927 21:06:22.839330 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.839362 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 21:06:22.839485 7443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:40177: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:22.845106 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.837553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.16.190.253:34919)
0927 21:06:22.837625 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.837627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.837638 (+ 11us) server_negotiation.cc:407] Connection header received
0927 21:06:22.837668 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.837669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.837683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.837762 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.838128 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.838772 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.839724 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.839957 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.840041 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.842333 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.842334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.842335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.842343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.843849 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.844249 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.844251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.844252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.844314 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.844525 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.844526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.844526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.844646 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.844726 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.844928 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.845008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 21:06:22.845106 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.837401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60182)
0927 21:06:22.837494 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.837509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.837602 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.837770 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.837771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.837952 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.838020 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.838022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.838869 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.838870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.839638 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.839641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.840041 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.840042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.840122 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.840662 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.840668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.842245 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.843976 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.843977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.843978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.844166 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.844391 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.844391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.844392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.844456 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.844715 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.844716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.844783 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.844902 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.845007 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 21:06:22.845681 7368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:22.846047 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.845940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46836)
0927 21:06:22.845991 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.846019 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 21:06:22.846393 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.846304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46852)
0927 21:06:22.846348 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.846371 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 21:06:22.846513 7368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:40177: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
I20260927 21:06:22.877692 7368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } attempt: 1
I20260927 21:06:22.878353 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.878122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46860)
0927 21:06:22.878226 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.878294 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 21:06:22.878914 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.878763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46862)
0927 21:06:22.878857 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.878889 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 21:06:22.879071 7368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:40177: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
W20260927 21:06:22.890439 7450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 21:06:22.890663 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 21:06:22.890681 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 21:06:22.890702 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 21:06:22.891825 7450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 21:06:22.891860 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 21:06:22.891875 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 21:06:22.891887 7450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 21:06:22.891899 7450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 21:06:22.893419 7450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.190.212:38513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.190.254:41161,127.16.190.253:34919,127.16.190.252:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/krb5kdc/kudu_127.16.190.252.keytab
--principal=kudu/127.16.190.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.190.252:40177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.190.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 20:10:07 UTC on ccycloud.quasar-pwomtc.root.comops.site
build id 4676316
I20260927 21:06:22.893621 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 21:06:22.893797 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 21:06:22.899257 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:22.899323 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 21:06:22.899276 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 21:06:22.899327 7450 server_base.cc:1034] running on GCE node
I20260927 21:06:22.899809 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 21:06:22.900213 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 21:06:22.901399 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790543182901390 us; error 52 us; skew 500 ppm
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 21:06:22.903967 7450 init.cc:377] Logged in from keytab as kudu/127.16.190.252@KRBTEST.COM (short username kudu)
I20260927 21:06:22.904487 7450 webserver.cc:460] Webserver started at http://127.16.190.252:38465/ using document root <none> and password file <none>
I20260927 21:06:22.904735 7450 fs_manager.cc:359] Metadata directory not provided
I20260927 21:06:22.904806 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 21:06:22.904980 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 21:06:22.906073 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data/instance:
uuid: "0e5de369fd0f43559e18e80b26ba297f"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "a2b917dcc478efc4b04b9970044338ae"
server_key_iv: "36e89245470591c732ad85bee47f9bb9"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.906462 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal/instance:
uuid: "0e5de369fd0f43559e18e80b26ba297f"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "a2b917dcc478efc4b04b9970044338ae"
server_key_iv: "36e89245470591c732ad85bee47f9bb9"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.907676 7443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } attempt: 2
I20260927 21:06:22.907960 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 21:06:22.908100 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.907955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46876)
0927 21:06:22.908023 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.908062 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 21:06:22.908533 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.908423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46882)
0927 21:06:22.908474 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.908505 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 21:06:22.908677 7443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.190.252:40177: Network error: Client connection negotiation failed: client connection to 127.16.190.252:40177: connect: Connection refused (error 111)
I20260927 21:06:22.908946 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 21:06:22.909132 7450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 21:06:22.909200 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data,/tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal
uuid: "0e5de369fd0f43559e18e80b26ba297f"
format_stamp: "Formatted at 2026-09-27 21:06:22 on dist-test-slave-6c7s"
server_key: "a2b917dcc478efc4b04b9970044338ae"
server_key_iv: "36e89245470591c732ad85bee47f9bb9"
server_key_version: "encryptionkey@0"
I20260927 21:06:22.909253 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 21:06:22.919198 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 21:06:22.921264 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 21:06:22.921376 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 21:06:22.925355 7450 rpc_server.cc:307] RPC server started. Bound to: 127.16.190.252:40177
I20260927 21:06:22.925410 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.190.252:40177 every 8 connection(s)
I20260927 21:06:22.925768 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/data/info.pb
I20260927 21:06:22.926728 7518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 21:06:22.926806 17147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQTa3tm/build/release/bin/kudu as pid 7450
I20260927 21:06:22.926934 17147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQTa3tm/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790542906842326-17147-0/minicluster-data/master-2/wal/instance
I20260927 21:06:22.927145 17147 external_mini_cluster.cc:1428] Setting key 88933df6ee52c5ee9a61b35a2e691284
I20260927 21:06:22.929283 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:06:22.938727 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.930060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.16.190.254:41161)
0927 21:06:22.930133 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.930135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.930311 (+ 176us) server_negotiation.cc:407] Connection header received
0927 21:06:22.930376 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.930376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.930386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.930433 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.930952 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.931392 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.932349 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.932525 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.932603 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.935373 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.935375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.935375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.935383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.937013 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.937509 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.937509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.937510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.937563 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.937873 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.937874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.937874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.938029 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.938107 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.938533 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.938606 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 21:06:22.939072 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.930019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.254:41161 (local address 127.0.0.1:39034)
0927 21:06:22.930202 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.930244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.930315 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.930468 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.930470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.930799 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.930882 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.930883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.931472 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.931473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.932263 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.932264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.932901 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.932902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.932976 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.933437 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.933449 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.935253 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.937097 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.937102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.937105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.937426 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.937627 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.937628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.937628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.937772 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.938105 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.938111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.938366 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.938496 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.938664 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
I20260927 21:06:22.939214 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.927689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.16.190.252:40177)
0927 21:06:22.927882 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.927891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.927928 (+ 37us) server_negotiation.cc:407] Connection header received
0927 21:06:22.928464 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.928474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.928544 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.928742 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.929455 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.930876 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.932106 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.932523 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.932640 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.935812 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.935822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.935822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.935830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.937805 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.938197 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.938200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.938201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.938263 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.938454 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.938455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.938456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.938614 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.938794 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.939040 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.939123 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260927 21:06:22.939625 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.254@KRBTEST.COM
I20260927 21:06:22.947276 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.939914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.253:34919 (local address 127.0.0.1:60184)
0927 21:06:22.939949 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.939959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.940022 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.940177 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.940178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.940335 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.940392 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.940393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.941184 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.941186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.941847 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.941850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.942366 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.942367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.942450 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.942990 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.942994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.944458 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.946128 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.946129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.946130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.946353 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.946596 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.946597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.946597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.946658 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.946931 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.946932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.946983 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.947065 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.947170 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 21:06:22.947332 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.939973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.16.190.253:34919)
0927 21:06:22.940045 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.940047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.940068 (+ 21us) server_negotiation.cc:407] Connection header received
0927 21:06:22.940094 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.940095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.940108 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.940161 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.940471 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.941111 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.941931 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.942241 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.942350 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.944562 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.944564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.944565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.944574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.946064 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.946455 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.946457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.946457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.946521 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.946731 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.946732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.946733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.946857 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.946928 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.947099 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.947171 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 21:06:22.947896 7518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } has no permanent_uuid. Determining permanent_uuid...
I20260927 21:06:22.948359 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.940070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.16.190.254:41161)
0927 21:06:22.940107 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.940108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.940128 (+ 20us) server_negotiation.cc:407] Connection header received
0927 21:06:22.940186 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.940186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.940196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.940249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.940867 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.941489 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.942401 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.942582 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.942636 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.945351 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.945353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.945353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.945359 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.946826 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.947187 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.947189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.947189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.947247 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.947458 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.947459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.947459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.947679 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.947817 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.948152 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.948253 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.252@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.190.253@KRBTEST.COM
I20260927 21:06:22.954281 7368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } attempt: 2
I20260927 21:06:22.955993 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.948234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46892)
0927 21:06:22.948390 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.948400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.948446 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.948567 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.948567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.948713 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.948782 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.948784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.949600 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.949601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.950342 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.950345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.950734 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.950735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.950782 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.951254 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.951259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.952643 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.954675 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.954676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.954677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.954845 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.955066 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.955067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.955067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.955120 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.955626 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.955627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.955663 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.955795 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.955891 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 21:06:22.957050 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.948378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.16.190.252:40177)
0927 21:06:22.948433 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.948435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.948443 (+ 8us) server_negotiation.cc:407] Connection header received
0927 21:06:22.948512 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.948513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.948521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.948562 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.948872 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.949519 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.950422 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.950648 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.950734 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.952737 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.952739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.952739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.952747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.954599 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.954939 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.954939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.954940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.955004 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.955180 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.955181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.955181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.955322 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.955384 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.956805 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.956940 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 21:06:22.958308 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.949336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.16.190.253:34919)
0927 21:06:22.949382 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.949384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.949434 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:06:22.949500 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.949501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.949508 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.949550 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.950207 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.950665 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.951564 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.951788 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.951892 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.954921 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.954923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.954923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.954930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.956528 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.956941 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.956942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.956943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.957002 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.957561 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.957562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.957562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.957700 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.957870 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.958056 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.958133 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:06:22.958973 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: Bootstrap starting.
I20260927 21:06:22.959662 7518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: Neither blocks nor log segments found. Creating new log.
Sep 27 21:06:22 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.254@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:06:22.959980 7518 log.cc:824] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:22.960178 17147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 21:06:22.961212 7518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: No bootstrap required, opened a new log
I20260927 21:06:22.961481 7342 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bdcd4220b7644c2da588f255fe1181fa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39050
I20260927 21:06:22.963115 7518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.963275 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.954589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46900)
0927 21:06:22.954648 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:22.954659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:22.954736 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:22.955434 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:22.955435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:22.955591 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:22.955661 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.955663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.956793 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.956795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.957680 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:22.957683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.958170 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.958171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:22.958234 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.958721 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:22.958726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:22.960026 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:22.962106 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.962107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.962107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.962349 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.962578 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:22.962579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:22.962579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:22.962646 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:22.962954 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:22.962955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:22.963003 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 21:06:22.963081 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 21:06:22.963179 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 21:06:22.963267 7518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5de369fd0f43559e18e80b26ba297f, State: Initialized, Role: FOLLOWER
I20260927 21:06:22.963359 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.954862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.16.190.252:40177)
0927 21:06:22.955056 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.955058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.955065 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:06:22.955084 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.955085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.955095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.955421 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.955759 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.956710 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.957770 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.958069 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.958448 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.960168 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.960169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.960169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.960176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.962009 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.962435 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.962436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.962436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.962500 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.962717 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.962718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.962718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.962882 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.962946 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.963117 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.963204 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
I20260927 21:06:22.963375 7518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.963685 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.963780 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:22.963729 7518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:22.964775 7368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa: Bootstrap starting.
I20260927 21:06:22.965507 7368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:22.965834 7368 log.cc:824] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa: Log is configured to *not* fsync() on all Append() calls
W20260927 21:06:22.966228 7536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:22.966290 7536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:22.966591 7368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa: No bootstrap required, opened a new log
I20260927 21:06:22.967983 7368 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.968122 7368 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcd4220b7644c2da588f255fe1181fa, State: Initialized, Role: FOLLOWER
I20260927 21:06:22.968214 7368 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.968557 7342 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 21:06:22.968703 7342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:22.966467 7450 master_runner.cc:186] Error getting master registration for 127.16.190.254:41161: OK, instance_id { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" instance_seqno: 1790543182609779 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:06:22.968525 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.968868 7450 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:06:22.968924 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:22.969543 7342 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.969803 7342 leader_election.cc:288] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [CANDIDATE]: Term 1 election: Requested vote from peers 5de926487e134f489861e9bd770ee8a4 (127.16.190.253:34919), 0e5de369fd0f43559e18e80b26ba297f (127.16.190.252:40177)
I20260927 21:06:22.970110 7492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdcd4220b7644c2da588f255fe1181fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e5de369fd0f43559e18e80b26ba297f"
I20260927 21:06:22.970225 7492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:22.971077 7492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd4220b7644c2da588f255fe1181fa in term 1.
I20260927 21:06:22.971496 7316 leader_election.cc:302] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5de369fd0f43559e18e80b26ba297f, bdcd4220b7644c2da588f255fe1181fa; no voters:
I20260927 21:06:22.971662 7539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 21:06:22.971827 7539 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [term 1 LEADER]: Becoming Leader. State: Replica: bdcd4220b7644c2da588f255fe1181fa, State: Running, Role: LEADER
I20260927 21:06:22.971918 7539 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:22.968547 7368 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:22.972355 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcd4220b7644c2da588f255fe1181fa. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.972572 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:22.974196 7549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 21:06:22.976567 7549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 21:06:22.975296 7300 master_runner.cc:186] Error getting master registration for 127.16.190.253:34919: OK, instance_id { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" instance_seqno: 1790543182763101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 21:06:22.977854 7300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 21:06:22.978988 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.970914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.16.190.254:41161)
0927 21:06:22.970960 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.970963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.971090 (+ 127us) server_negotiation.cc:407] Connection header received
0927 21:06:22.971199 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.971200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.971210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.971240 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.972043 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.972717 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.973895 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.974240 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.974288 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.975843 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.975845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.975845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.975852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.977491 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.977894 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.977895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.977896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.977952 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.978142 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.978143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.978143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.978266 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.978358 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.978812 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.978893 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:06:22.983881 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.971414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.16.190.252:40177)
0927 21:06:22.971637 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.971639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.973731 (+ 2092us) server_negotiation.cc:407] Connection header received
0927 21:06:22.973758 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.973759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.973770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.973831 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.975183 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.976129 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.977609 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.977943 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.978015 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.980306 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.980308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.980308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.980316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.981991 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.982725 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.982727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.982727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.982791 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.983086 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.983087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.983087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.983247 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.983325 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.983648 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.983784 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260927 21:06:22.984943 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:22.971453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.16.190.253:34919)
0927 21:06:22.971497 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:22.971499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:22.973094 (+ 1595us) server_negotiation.cc:407] Connection header received
0927 21:06:22.973183 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:22.973184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:22.973196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:22.973225 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:22.974938 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.975719 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.978619 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:22.979418 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:22.979462 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:22.981666 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:22.981668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:22.981668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:22.981677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:22.983416 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.983815 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.983816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.983816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.983874 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:22.984331 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:22.984332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:22.984332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:22.984464 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:22.984555 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:22.984750 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 21:06:22.984854 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 21:06:22.986639 7492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [term 1 FOLLOWER]: Refusing update from remote peer bdcd4220b7644c2da588f255fe1181fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:06:22.987067 7547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:06:22.988497 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcd4220b7644c2da588f255fe1181fa. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.988611 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:22.989606 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.989727 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: This master's current role is: LEADER
I20260927 21:06:22.989848 7525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:22.989919 7525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:22.990643 7563 mvcc.cc:205] Tried to move back new op lower bound from 7334064877510463488 to 7334064877453611008. Current Snapshot: MvccSnapshot[applied={T|T < 7334064877510463488}]
I20260927 21:06:22.990794 7549 catalog_manager.cc:1347] Generated new cluster ID: 1d0883098b9a438ba635957bfa2f4b6f
I20260927 21:06:22.990845 7549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 21:06:22.999809 7443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } attempt: 3
Sep 27 21:06:23 dist-test-slave-6c7s krb5kdc[7287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790543182, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.190.253@KRBTEST.COM for kudu/127.16.190.252@KRBTEST.COM
I20260927 21:06:23.010725 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:23.000163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.190.252:40177 (local address 127.0.0.1:46918)
0927 21:06:23.000338 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 21:06:23.000351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 21:06:23.000397 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 21:06:23.000644 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 21:06:23.000645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 21:06:23.000877 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 21:06:23.000929 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:23.000931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:23.002771 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:23.002772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:23.003608 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 21:06:23.003611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 21:06:23.004198 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 21:06:23.004199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 21:06:23.004284 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:23.004896 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 21:06:23.004902 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 21:06:23.006451 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 21:06:23.009494 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:23.009495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:23.009496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:23.009732 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:23.009986 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 21:06:23.009987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 21:06:23.009987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 21:06:23.010060 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 21:06:23.010397 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 21:06:23.010398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 21:06:23.010455 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 21:06:23.010509 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 21:06:23.010614 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 21:06:23.011392 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:23.000470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.16.190.252:40177)
0927 21:06:23.000509 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.000510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.000517 (+ 7us) server_negotiation.cc:407] Connection header received
0927 21:06:23.000543 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.000544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.000553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.000597 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.001016 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:23.001808 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:23.003728 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:23.004056 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:23.004118 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:23.006545 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 21:06:23.006546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 21:06:23.006547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 21:06:23.006554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 21:06:23.008465 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:23.009830 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:23.009831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:23.009832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:23.009904 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 21:06:23.010139 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 21:06:23.010139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 21:06:23.010140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 21:06:23.010297 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 21:06:23.010354 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:23.011138 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 21:06:23.011229 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 21:06:23.013577 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: Bootstrap starting.
I20260927 21:06:23.014361 7443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: Neither blocks nor log segments found. Creating new log.
I20260927 21:06:23.014664 7443 log.cc:824] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: Log is configured to *not* fsync() on all Append() calls
I20260927 21:06:23.015530 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: No bootstrap required, opened a new log
I20260927 21:06:23.017742 7443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:23.017900 7443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de926487e134f489861e9bd770ee8a4, State: Initialized, Role: FOLLOWER
I20260927 21:06:23.017984 7443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } }
I20260927 21:06:23.018332 7443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 21:06:23.018533 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bdcd4220b7644c2da588f255fe1181fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5de926487e134f489861e9bd770ee8a4"
I20260927 21:06:23.019619 7549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 21:06:23.020025 7568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:23.020423 7549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 21:06:23.020241 7416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 21:06:23.021486 7416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [term 1 FOLLOWER]: Refusing update from remote peer bdcd4220b7644c2da588f255fe1181fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 21:06:23.022048 7539 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 21:06:23.023010 7417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdcd4220b7644c2da588f255fe1181fa in term 1.
I20260927 21:06:23.023272 7568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:23.024304 7581 mvcc.cc:205] Tried to move back new op lower bound from 7334064877623861248 to 7334064877453611008. Current Snapshot: MvccSnapshot[applied={T|T < 7334064877510463488}]
I20260927 21:06:23.024600 7568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcd4220b7644c2da588f255fe1181fa. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:23.024673 7568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 21:06:23.024701 7568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:23.024734 7568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 21:06:23.029465 7579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 21:06:23.029577 7579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 21:06:23.030395 7549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa: Generated new TSK 0
I20260927 21:06:23.030560 7549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 21:06:23.037298 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bdcd4220b7644c2da588f255fe1181fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdcd4220b7644c2da588f255fe1181fa" member_type: VOTER last_known_addr { host: "127.16.190.254" port: 41161 } } peers { permanent_uuid: "5de926487e134f489861e9bd770ee8a4" member_type: VOTER last_known_addr { host: "127.16.190.253" port: 34919 } } peers { permanent_uuid: "0e5de369fd0f43559e18e80b26ba297f" member_type: VOTER last_known_addr { host: "127.16.190.252" port: 40177 } } }
I20260927 21:06:23.037427 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcd4220b7644c2da588f255fe1181fa [sys.catalog]: This master's current role is: LEADER
W20260927 21:06:23.053805 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.052286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.16.190.253:34919)
0927 21:06:23.052332 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.052335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.053071 (+ 736us) server_negotiation.cc:407] Connection header received
0927 21:06:23.053172 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.053173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.053185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.053215 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.053730 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60214: BlockingRecv error: recv got EOF from 127.0.0.1:60214 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 21:06:23.054613 7372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.052687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.16.190.254:41161)
0927 21:06:23.052734 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.052736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.054075 (+ 1339us) server_negotiation.cc:407] Connection header received
0927 21:06:23.054174 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.054175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.054187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.054216 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.054532 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingRecv error: recv got EOF from 127.0.0.1:39080 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:06:23.055704 7523 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.054988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.16.190.252:40177)
0927 21:06:23.055042 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.055044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.055142 (+ 98us) server_negotiation.cc:407] Connection header received
0927 21:06:23.055238 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.055238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.055248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.055277 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.055629 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46930: BlockingRecv error: recv got EOF from 127.0.0.1:46930 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 21:06:23.062142 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.061237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.16.190.253:34919)
0927 21:06:23.061301 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.061303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.061510 (+ 207us) server_negotiation.cc:407] Connection header received
0927 21:06:23.061631 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.061632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.061669 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.061700 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.062080 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60216: BlockingRecv error: recv got EOF from 127.0.0.1:60216 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 21:06:23.065416 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.064550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.16.190.253:34919)
0927 21:06:23.064587 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.064589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.064879 (+ 290us) server_negotiation.cc:407] Connection header received
0927 21:06:23.064980 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.064981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.064993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.065021 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.065355 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60222: BlockingRecv error: recv got EOF from 127.0.0.1:60222 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 21:06:23.069450 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.068552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.16.190.253:34919)
0927 21:06:23.068592 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.068594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.068883 (+ 289us) server_negotiation.cc:407] Connection header received
0927 21:06:23.068978 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.068979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.068991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.069019 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.069395 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60224: BlockingRecv error: recv got EOF from 127.0.0.1:60224 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 21:06:23.075726 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.074804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.16.190.253:34919)
0927 21:06:23.074845 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.074847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.075137 (+ 290us) server_negotiation.cc:407] Connection header received
0927 21:06:23.075246 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.075247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.075259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.075290 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.075668 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60228: BlockingRecv error: recv got EOF from 127.0.0.1:60228 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:06:23.086081 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.085070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.16.190.253:34919)
0927 21:06:23.085118 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.085120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.085425 (+ 305us) server_negotiation.cc:407] Connection header received
0927 21:06:23.085537 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.085539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.085551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.085583 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.086003 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60236: BlockingRecv error: recv got EOF from 127.0.0.1:60236 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 21:06:23.104367 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.103386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.16.190.253:34919)
0927 21:06:23.103431 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.103433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.103745 (+ 312us) server_negotiation.cc:407] Connection header received
0927 21:06:23.103856 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.103858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.103871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.103903 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.104306 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60246: BlockingRecv error: recv got EOF from 127.0.0.1:60246 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 21:06:23.138717 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.137775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.16.190.253:34919)
0927 21:06:23.137821 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.137823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.138098 (+ 275us) server_negotiation.cc:407] Connection header received
0927 21:06:23.138201 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.138202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.138215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.138244 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.138635 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60254: BlockingRecv error: recv got EOF from 127.0.0.1:60254 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 21:06:23.204952 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.204242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.16.190.253:34919)
0927 21:06:23.204299 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.204301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.204385 (+ 84us) server_negotiation.cc:407] Connection header received
0927 21:06:23.204418 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.204419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.204433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.204518 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.204888 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60258: BlockingRecv error: recv got EOF from 127.0.0.1:60258 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 21:06:23.335060 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.334204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.16.190.253:34919)
0927 21:06:23.334306 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.334309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.334422 (+ 113us) server_negotiation.cc:407] Connection header received
0927 21:06:23.334524 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.334525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.334544 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.334605 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.334978 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60270: BlockingRecv error: recv got EOF from 127.0.0.1:60270 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 21:06:23.593102 7447 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 21:06:23.592341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.16.190.253:34919)
0927 21:06:23.592413 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:23.592415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:23.592448 (+ 33us) server_negotiation.cc:407] Connection header received
0927 21:06:23.592569 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:23.592570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:23.592580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:23.592635 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 21:06:23.593032 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60278: BlockingRecv error: recv got EOF from 127.0.0.1:60278 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 21:06:23.966758 7536 catalog_manager.cc:1259] Loaded cluster ID: 1d0883098b9a438ba635957bfa2f4b6f
I20260927 21:06:23.966840 7536 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: loading cluster ID for follower catalog manager: success
I20260927 21:06:23.967768 7536 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: acquiring CA information for follower catalog manager: success
I20260927 21:06:23.968132 7536 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e5de369fd0f43559e18e80b26ba297f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:06:24.030128 7579 catalog_manager.cc:1259] Loaded cluster ID: 1d0883098b9a438ba635957bfa2f4b6f
I20260927 21:06:24.030225 7579 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: loading cluster ID for follower catalog manager: success
I20260927 21:06:24.031141 7579 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: acquiring CA information for follower catalog manager: success
I20260927 21:06:24.031525 7579 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5de926487e134f489861e9bd770ee8a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 21:06:24.109041 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:24.106377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.16.190.253:34919)
0927 21:06:24.106488 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:24.106489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:24.106499 (+ 10us) server_negotiation.cc:407] Connection header received
0927 21:06:24.106574 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:24.106575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:24.106625 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:24.106704 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:24.106836 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.107328 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.108118 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.108333 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.108391 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:24.108561 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:24.108719 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:24.108785 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:24.108882 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 21:06:24.108912 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260927 21:06:24.113616 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:24.110995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.16.190.252:40177)
0927 21:06:24.111110 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:24.111112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:24.111191 (+ 79us) server_negotiation.cc:407] Connection header received
0927 21:06:24.111234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:24.111235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:24.111280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:24.111365 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:24.111465 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.112034 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.112767 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.113038 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.113096 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:24.113264 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:24.113383 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:24.113473 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:24.113496 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 21:06:24.113516 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260927 21:06:24.118587 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:24.115252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.16.190.253:34919)
0927 21:06:24.115298 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:24.115301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:24.115630 (+ 329us) server_negotiation.cc:407] Connection header received
0927 21:06:24.115751 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:24.115753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:24.115762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:24.115814 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:24.115917 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.116420 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.117639 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.117955 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.118011 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:24.118139 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:24.118173 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:24.118233 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:24.118419 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 21:06:24.118450 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 21:06:24.118780 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:24.115215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.16.190.254:41161)
0927 21:06:24.115438 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:24.115440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:24.115495 (+ 55us) server_negotiation.cc:407] Connection header received
0927 21:06:24.115577 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:24.115578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:24.115625 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:24.115753 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:24.116024 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.116716 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.117567 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.117900 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.117968 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:24.118315 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:24.118434 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:24.118495 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:24.118667 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 21:06:24.118694 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":63,"threads_started":1}
I20260927 21:06:24.119163 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 21:06:24.115279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.16.190.252:40177)
0927 21:06:24.115405 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 21:06:24.115407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 21:06:24.115457 (+ 50us) server_negotiation.cc:407] Connection header received
0927 21:06:24.115522 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 21:06:24.115523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 21:06:24.115535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 21:06:24.115590 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 21:06:24.115745 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.116486 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.118041 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 21:06:24.118377 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 21:06:24.118435 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 21:06:24.118823 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 21:06:24.118944 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 21:06:24.119016 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 21:06:24.119068 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 21:06:24.119097 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 21:06:24.124593 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 7300
I20260927 21:06:24.131004 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 7375
I20260927 21:06:24.136525 17147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQTa3tm/build/release/bin/kudu with pid 7450
2026-09-27T21:06:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2827 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5692 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277301 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST